Skip to main content

Using GitOps + KubeVela for Application Continuous Delivery

· 13 min read
Tianxin Dong

KubeVela is a simple, easy-to-use, and highly extensible cloud-native application platform. It can make developers deliver microservices applications easily, without knowing Kubernetes details.

KubeVela is based on OAM model, which naturally solves the orchestration problems of complex resources. It means that KubeVela can manage complex large-scale applications with GitOps. Convergence of team and system size after the system complexity problem.

What is GitOps​

GitOps is a modern way to do continuous delivery. Its core idea is to have a Git repository which contains environmental and application configurations. An automated process is also needed for sync the config to cluster.

By changing the files in repository, developers can apply the applications automatically. The benefits of applying GitOps include:

  • Increased productivity. Continuous delivery can speed up the time of deployment.
  • Lower the barrier for developer to deploy. By pushing code instead of container configuration, developers can easily deploy Kubernetes without knowing its internal implementation.
  • Trace the change records. Managing the cluster with Git makes every change traceable, enhancing the audit trail.
  • Recover the cluster with Git's rollback and branch.

GitOps with KubeVela​

KubeVela as an declarative application delivery control plane can be naturally used in GitOps approach, and this will provide below extra bonus to end users alongside with GitOps benefits:

  • application delivery workflow (CD pipeline)
    • i.e. KubeVela supports pipeline style application delivery process in GitOps, instead of simply declaring final status;
  • handling deployment dependencies and designing typologies (DAG);
  • unified higher level abstraction atop various GitOps tools' primitives;
  • declare, provision and consume cloud resources in unified application definition;
  • various out-of-box deployment strategies (Canary, Blue-Green ...);
  • various out-of-box hybrid/multi-cloud deployment policies (placement rule, cluster selectors etc.);
  • Kustomize-style patch for multi-env deployment without the need to learn Kustomize at all;
  • ... and much more.

In this section, we will introduce steps of using KubeVela directly in GitOps approach.

GitOps workflow​

The GitOps workflow is divided into CI and CD:

  • CI(Continuous Integration): Continuous integration builds code and images, and pushes images to the registry. There are many CI tools like GitHub Action, Travis, Jenkins and so on. In this article, we use GitHub Action for CI. You can also use other CI tools. KubeVela can connect CI processes under any tool around GitOps.
  • CD(Continuous Delivery): Continuous delivery automatically updates the configuration in the cluster. For example, update the latest images in the registry to the cluster.
    • Currently there are two main CD modes:
      • Push-based: Push mode CD is mainly accomplished by configuring CI pipeline. In this way, the access key of the cluster is shared with CI so that the CI pipeline can push changes to the cluster. For this mode, please refer to our previous blog post: Using Jenkins + KubeVela for Application Continuous Delivery.
      • Pull-based: Pull mode CD listens for changes to the repository (code repository or configuration repository) in the cluster and synchronizes those changes to the cluster. In this way, the cluster actively pulls the update, thus avoiding the problem of exposing the secret key. This article will introduce using KubeVela and GitOps in pull mode.

This article will separate into two perspectives:

  1. For platform administrators/SREs, they can update the config in Git repo. It will trigger automated re-deployment.

  2. For developers, they can update the app source code and then push it to Git. It will trigger building latest image and re-deployment.

For platform administrators/SREs​

Platform administrators/SREs prepares the Git repo for operational config. Every config change will be traceable by that. KubeVela will watch the repo and apply changes to the clusters.

alt

Setup Config Repository​

The configuration files are from the Example Repo.

In this example, we will deploy an application and a database, the application uses the database to store data.

The structure of the config repository looks below:

  • The clusters/ contains the GitOps config. It will command KubeVela to watch the specified repo and apply latest changes.
  • The apps/ contains the Application yaml for deploying the user-facing app.
  • The infrastructure/ contains infrastructure tools, i.e. MySQL database.
├── apps
│   └── my-app.yaml
├── clusters
│   ├── apps.yaml
│   └── infra.yaml
└── infrastructure
    └── mysql.yaml

KubeVela recommends using the directory structure above to manage your GitOps repository. clusters/ holds the associated KubeVela GitOps configuration that need to be applied to cluster manually, apps/ holds your application and infrastructure/ holds your base configuration. By separating applications from basic configurations, you can manage your deployment environment more reasonably and isolate application changes.

Directory clusters/​

The clusters/ is the initialize configuration directory for KubeVela GitOps.

Below is how the clusters/infra.yaml looks like:

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: infra
spec:
  components:
  - name: database-config
    type: kustomize
    properties:
      repoType: git
      # replace it with your repo url
      url: https://github.com/FogDong/KubeVela-GitOps-Infra-Demo
      # replace it with your git secret if it's a private repo
      # secretRef: git-secret
      # the pull interval time, set to 10m since the infrastructure is steady
      pullInterval: 10m
      git:
        # the branch name
        branch: main
      # the path to sync
      path: ./infrastructure

apps.yaml and infra.yaml in clusters/ are similar. Their difference is to watch different directories. In apps.yaml, the properties.path will be ./apps.

Apply the files in clusters/ manually. They will sync the files in infrastructure/ and apps/ dir of the Git repo.

Directory apps/​

The file in apps/ is a simple application with database information and Ingress. The app serves HTTP service and connects to a MySQL database. In the '/' path, it will display the version in the code; in the /db path, it will list the data in database.

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: my-app
  namespace: default
spec:
  components:
    - name: my-server
      type: webservice
      properties:
        image: <your image address> # {"$imagepolicy": "default:apps"}
        port: 8088
        env:
          - name: DB_HOST
            value: mysql-cluster-mysql.default.svc.cluster.local:3306
          - name: DB_PASSWORD
            valueFrom:
              secretKeyRef:
                name: mysql-secret
                key: ROOT_PASSWORD
      traits:
        - type: ingress
          properties:
            domain: testsvc.example.com
            http:
              /: 8088

This is an Application binds with Traits Ingress. In this way, the underlying Deployment, Service, and Ingress can be brought together in a single file, making it easier to manage the application.

Directory infrastructure/​

The infrastructure/ contains the config of some infrastructures like database. In the following, we will use MySQL operator to deploy a MySQL cluster.

Notice that there must be a secret in your cluster with MySQL password specified in key ROOT_PASSWORD.

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: mysql
  namespace: default
spec:
  components:
    - name: mysql-controller
      type: helm
      properties:
        repoType: helm
        url: https://presslabs.github.io/charts
        chart: mysql-operator
        version: "0.4.0"
    - name: mysql-cluster
      type: raw
      dependsOn:
        - mysql-controller
      properties:
        apiVersion: mysql.presslabs.org/v1alpha1
        kind: MysqlCluster
        metadata:
          name: mysql-cluster
        spec:
          replicas: 1
          # replace it with your secret
          secretName: mysql-secret

We use workflow in this Application. The first step is to deploy the MySQL controller, after the controller is running, the second step will deploy the MySQL cluster.

Apply the files in clusters/​

After storing bellow files in the Git config repo, we need to apply the GitOps config files in clusters/ manually.

First, apply the clusters/infra.yaml to cluster, we can see that the MySQL in infrastructure/ is automatically deployed:

kubectl apply -f clusters/infra.yaml
$ vela ls

APP     COMPONENT           TYPE        TRAITS  PHASE   HEALTHY STATUS  CREATED-TIME
infra   database-config     kustomize           running healthy         2021-09-26 20:48:09 +0800 CST
mysql   mysql-controller    helm                running healthy         2021-09-26 20:48:11 +0800 CST
└─      mysql-cluster       raw                 running healthy         2021-09-26 20:48:11 +0800 CST

Apply the clusters/apps.yaml to cluster, we can see that the application in apps/ is automatically deployed:

kubectl apply -f clusters/apps.yaml
APP     COMPONENT           TYPE        TRAITS  PHASE   HEALTHY STATUS  CREATED-TIME
apps    apps                kustomize           running healthy         2021-09-27 16:55:53 +0800 CST
infra   database-config     kustomize           running healthy         2021-09-26 20:48:09 +0800 CST
my-app  my-server           webservice  ingress running healthy         2021-09-27 16:55:55 +0800 CST
mysql   mysql-controller    helm                running healthy         2021-09-26 20:48:11 +0800 CST
└─      mysql-cluster       raw                 running healthy         2021-09-26 20:48:11 +0800 CST

By deploying the KubeVela GitOps config files, we now automatically apply the application and database in cluster.

curl the Ingress of the app, we can see that the current version is 0.1.5 and the application is connected to the database successfully:

$ kubectl get ingress
NAME        CLASS    HOSTS                 ADDRESS         PORTS   AGE
my-server   <none>   testsvc.example.com   <ingress-ip>    80      162m

$ curl -H "Host:testsvc.example.com" http://<ingress-ip>
Version: 0.1.5

$ curl -H "Host:testsvc.example.com" http://<ingress-ip>/db
User: KubeVela
Description: It's a test user

Modify the config for GitOps trigger​

After the first deployment, we can modify the files in config repo to update the applications in the cluster.

Modify the domain of the application's Ingress:

...
      traits:
        - type: ingress
          properties:
            domain: kubevela.example.com
            http:
              /: 8089

Check the Ingress in cluster after a while:

NAME        CLASS    HOSTS                 ADDRESS         PORTS   AGE
my-server   <none>   kubevela.example.com  <ingress-ip>    80      162m

The host of the Ingress has been updated successfully!

In this way, we can edit the files in the Git repo to update the cluster.

For developers​

Developers writes the application source code and push it to a Git repo (aka app repo). Once app repo updates, the CI will build the image and push it to the image registry. KubeVela watches the image registry, and updates the image in config repo. Finally, it will apply the config to the cluster.

User can update the configuration in the cluster automatically when the code is updated.

alt

Setup App Code Repository​

Setup a Git repository with source code and Dockerfile.

The app serves HTTP service and connects to a MySQL database. In the '/' path, it will display the version in the code; in the /db path, it will list the data in database.

    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        _, _ = fmt.Fprintf(w, "Version: %s\n", VERSION)
    })
    http.HandleFunc("/db", func(w http.ResponseWriter, r *http.Request) {
        rows, err := db.Query("select * from userinfo;")
        if err != nil {
            _, _ = fmt.Fprintf(w, "Error: %v\n", err)
        }
        for rows.Next() {
            var username string
            var desc string
            err = rows.Scan(&username, &desc)
            if err != nil {
                _, _ = fmt.Fprintf(w, "Scan Error: %v\n", err)
            }
            _, _ = fmt.Fprintf(w, "User: %s \nDescription: %s\n\n", username, desc)
        }
    })

    if err := http.ListenAndServe(":8088", nil); err != nil {
        panic(err.Error())
    }

In this tutorial, we will setup a CI pipeline using GitHub Actions to build the image and push it to a registry. The code and configuration files are from the Example Repo.

Create Git Secret for KubeVela committing to Config Repo​

After the new image is pushed to the image registry, KubeVela will be notified and update the Application file in the Git repository and cluster. Therefore, we need a secret with Git information for KubeVela to commit to the Git repository. Fill the following yaml files with your password and apply it to the cluster:

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
type: kubernetes.io/basic-auth
stringData:
  username: <your username>
  password: <your password>

Setup Config Repository​

The configuration repository is almost the same as the previous configuration, you only need to add the image registry config to the file. For more details, please refer to Example Repository.

Add the config of image registry in clusters/apps.yaml, it listens for image updates in the image registry:

...
  imageRepository:
    image: <your image>
    # if it's a private image registry, use `kubectl create secret docker-registry` to create the secret
    # secretRef: imagesecret
    filterTags:
      # filter the image tag
      pattern: '^master-[a-f0-9]+-(?P<ts>[0-9]+)'
      extract: '$ts'
    # use the policy to sort the latest image tag and update
    policy:
      numerical:
        order: asc
    # add more commit message
    commitMessage: "Image: {{range .Updated.Images}}{{println .}}{{end}}"

Modify the image field in apps/my-app.yaml and add annotation # {"$imagepolicy": "default:apps"}. Notice that KubeVela will only be able to modify the image field if the annotation is added after the field. default:apps is namespace:name of the GitOps config file above.

spec:
  components:
    - name: my-server
      type: webservice
      properties:
        image: ghcr.io/fogdong/test-fog:master-cba5605f-1632714412 # {"$imagepolicy": "default:apps"}

After update the files in clusters/ to cluster, we can then update the application by modifying the code.

Modify the code​

Change the Version to 0.1.6 and modify the data in database:

const VERSION = "0.1.6"

...

func InsertInitData(db *sql.DB) {
    stmt, err := db.Prepare(insertInitData)
    if err != nil {
        panic(err)
    }
    defer stmt.Close()

    _, err = stmt.Exec("KubeVela2", "It's another test user")
    if err != nil {
        panic(err)
    }
}

Commit the change to the Git Repository, we can see that our CI pipelines has built the image and push it to the image registry.

KubeVela will listen to the image registry and update the apps/my-app.yaml in Git Repository with the latest image tag.

We can see that there is a commit form kubevelabot, the commit message is always with a prefix Update image automatically. You can use format like {{range .Updated.Images}}{{println .}}{{end}} to specify the image name in the commitMessage field.

alt

Note that if you want to put the code and config in the same repository, you need to filter out the commit from KubeVela in CI configuration like below to avoid the repeat build of pipeline.

jobs:
 publish:
   if: "!contains(github.event.head_commit.message, 'Update image automatically')"

Re-check the Application in cluster, we can see that the image of the my-app has been updated after a while.

KubeVela polls the latest information from the code and image repo periodically (at an interval that can be customized):

  • When the Application file in the Git repository is updated, KubeVela will update the Application in the cluster based on the latest configuration.
  • When a new tag is added to the image registry, KubeVela will filter out the latest tag based on your policy and update it to Git repository. When the files in the repository are updated, KubeVela repeats the first step and updates the files in the cluster, thus achieving automatic deployment.

We can curl to Ingress to see the current version and data:

$ kubectl get ingress
NAME        CLASS    HOSTS                 ADDRESS         PORTS   AGE
my-server   <none>   kubevela.example.com  <ingress-ip>    80      162m

$ curl -H "Host:kubevela.example.com" http://<ingress-ip>
Version: 0.1.6

$ curl -H "Host:kubevela.example.com" http://<ingress-ip>/db
User: KubeVela
Description: It's a test user

User: KubeVela2
Description: It's another test user

The Version has been updated successfully! Now we're done with everything from changing the code to automatically applying to the cluster.

Summary​

For platform admins/SREs, they update the config repo to operate the application and infrastructure. KubeVela will synchronize the config to the cluster, simplifying the deployment process.

For end users/developers, they write the source code, push it to Git, and then re-deployment will happen. It will make CI to build the image. KubeVela will then update the image field and apply the deployment config.

By integrating with GitOps, KubeVela helps users speed up deployment and simplify continuous deployment.