Skip to main content

Using Jenkins + KubeVela for Application Continuous Delivery

Da Yin, Yang Song

Da Yin, Yang Song

KubeVela Team

KubeVela bridges the gap between applications and infrastructures, enabling easy delivery and management of development codes. Compared to Kubernetes objects, the Application in KubeVela better abstracts and simplifies the configurations which developers care about, and leave complex infrastruature capabilities and orchestration details to platform engineers. The KubeVela apiserver further exposes HTTP interfaces, which help developers to deploy applications even without Kubernetes cluster access.

This article will use Jenkins, a popular continuous integration tool, as basis and give a brief introduction to how to build GitOps-based application continuous delivery highway.

Continuous Delivery Highway#

As application developer, you might care more about whether your application is functioning correctly and if development is convenient. There will be several system components on this highway to help you achieve that.

  1. First, you need a git repo to place program codes, test codes and a YAML file to declare your KubeVela application.
  2. Second, you also need a continuous integration tool to help you automate the integration test of codes, build container images and push images to image repo.
  3. Finally, you need to have a Kubernetes cluster and install KubeVela in it, with its apiserver function enabled.

Currently, the access management for KubeVela apiserver is under construction. You will need to configure apiserver access in later version of KubeVela (after v1.1).

In this article, we adopt GitHub as the git repo, Jenkins as the CI tool, DockerHub as the image repo. We use a simple HTTP Server written in Golang as example. The whole process of continuous delivery is shown as below. We can see that on this highway of continuous delivery, developers only need to care about application development and managing code version with Git. The highway will help developer run integration test and deploy applications into target Kubernetes cluster automatically.

arch

Set-up Environment#

Jenkins#

This article takes Jenkins as the CI tool. Developers can choose other CI tools like Travis or GitHub Action.

First you need to set up Jenkins to deploy CI pipelines. The installation and initialization of Jenkins could refer to the official docs.

Notice that since the CI pipeline in this example is based on Docker and GitHub, you need to install related plugins in Jenkins (Dashboard > Manage Jenkins > Manage Plugins), including Pipeline、HTTP Request Plugin、Docker Pipeline、Docker Plugin.

Besides, you also need to configure Docker environment for Jenkins to use (Dashboard > Manage Jenkins > Configure System > Docker Builder). If Docker has already been installed, you can set Docker URL as unix:///var/run/docker.sock.

Since the docker image will be pushed to image repo during the running of CI pipelines, you also need to store image repo accounts in Jenkins Credintial (Dashboard > Manage Jenkins > Manage Credentials > Add Credentials), such as DockerHub username and password.

jenkins-credential

GitHub#

This example uses GitHub as git repo. Developer can change it to other repos on demand, such as Gitlab.

To enable Jenkins to retrieve GitHub updates and write pipeline status back to GitHub, you need to execute the following two steps in GitHub.

  1. Configure Personal Access Token. Notice to check repo:status to get the permission for writing commit status.

github-pat

Then fill Personal Access Token from GitHub in Jenkins Credential (with Secret Text type).

jenkins-secret-text

Finally, go to Dashboard > Manage Jenkins > Configure System > GitHub in Jenkins and click Add GitHub Server to fill the newly created credential in. You can click Test connection to check if the configuration is correct.

jenkins-github

  1. Add Webhook to GitHub code repo settings. Fill Jenkins Webhook address into it. For example, http://my-jenkins.example.com/github-webhook/ . In this way, all Push events in this code repo will be pushed to Jenkins.

github-webhook

KubeVela#

You need to install KubeVela in your Kubernetes cluster and enable the apiserver function. Refer to official doc for details.

Composing Applications#

We use a simple HTTP Server as example. Here, we declare a constant named VERSION and print it when accessing the HTTP service. A simple test is also set up, which can be used to validate the format of VERSION.

// main.go
package main
import (
"fmt"
"net/http"
)
const VERSION = "0.1.0-v1alpha1"
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
_, _ = fmt.Fprintf(w, "Version: %s\n", VERSION)
})
if err := http.ListenAndServe(":8088", nil); err != nil {
println(err.Error())
}
}
// main_test.go
package main
import (
"regexp"
"testing"
)
const verRegex string = `^v?([0-9]+)(\.[0-9]+)?(\.[0-9]+)?` +
`(-([0-9A-Za-z\-]+(\.[0-9A-Za-z\-]+)*))?` +
`(\+([0-9A-Za-z\-]+(\.[0-9A-Za-z\-]+)*))?$`
func TestVersion(t *testing.T) {
if ok, _ := regexp.MatchString(verRegex, VERSION); !ok {
t.Fatalf("invalid version: %s", VERSION)
}
}

To build container image for the HTTP server and publishing it as KubeVela Application into Kubernetes, we also need another two files Dockerfile and app.yaml in the code repo. They are used to describe how container image is built and configure the KubeVela Application respectively.

# Dockerfile
FROM golang:1.13-rc-alpine3.10 as builder
WORKDIR /app
COPY main.go .
RUN go build -o kubevela-demo-cicd-app main.go
FROM alpine:3.10
WORKDIR /app
COPY --from=builder /app/kubevela-demo-cicd-app /app/kubevela-demo-cicd-app
ENTRYPOINT ./kubevela-demo-cicd-app
EXPOSE 8088

In app.yaml, we declare the application should contain 5 replica and expose the service through Ingress. The labels trait is used to tag Application Pods with current git commit id. Then the delivery pipeline in Jenkins will inject GIT_COMMIT into it and submit the Application configuration to KubeVela apiserver. Then the updates for Application will be triggered. The application will update 2 replica first, then hang and wait for manual approve. After developer confirms the change is valid, the rest 3 replica will be updated. This canary release is configured by the rollout trait declared in the Application.

# app.yaml
apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
name: kubevela-demo-app
spec:
components:
- name: kubevela-demo-app-web
type: webservice
properties:
image: somefive/kubevela-demo-cicd-app
imagePullPolicy: Always
port: 8080
traits:
- type: rollout
properties:
rolloutBatches:
- replicas: 2
- replicas: 3
batchPartition: 0
targetSize: 5
- type: labels
properties:
jenkins-build-commit: GIT_COMMIT
- type: ingress
properties:
domain: kubevela-demo-cicd-app.cf7c0ed25b151437ebe1ef58efc29bca4.us-west-1.alicontainer.com
http:
"/": 8088

Configure CI pipelines#

In this article, we set up two pipelines in Jenkins. One is the test pipeline, which is for running tests for application codes. The other one is the delivery pipeline, which builds container images and uploads them to image repo. Then the application configuration will be updated.

Test Pipeline#

Create a new pipeline in Jenkins. Set Build Triggers as GitHub hook trigger for GITScm polling.

test-pipeline-create test-pipeline-config

This pipeline uses golang image as execution environment at first. Next, it checkouts the dev branch of the target GitHub repo, indicating that this pipeline will be triggered by push events to dev branch. The piepline status will be written back to GitHub after execution finished.

void setBuildStatus(String message, String state) {
step([
$class: "GitHubCommitStatusSetter",
reposSource: [$class: "ManuallyEnteredRepositorySource", url: "https://github.com/Somefive/KubeVela-demo-CICD-app"],
contextSource: [$class: "ManuallyEnteredCommitContextSource", context: "ci/jenkins/test-status"],
errorHandlers: [[$class: "ChangingBuildStatusErrorHandler", result: "UNSTABLE"]],
statusResultSource: [ $class: "ConditionalStatusResultSource", results: [[$class: "AnyBuildResult", message: message, state: state]] ]
]);
}
pipeline {
agent {
docker { image 'golang:1.13-rc-alpine3.10' }
}
stages {
stage('Prepare') {
steps {
script {
def checkout = git branch: 'dev', url: 'https://github.com/Somefive/KubeVela-demo-CICD-app.git'
env.GIT_COMMIT = checkout.GIT_COMMIT
env.GIT_BRANCH = checkout.GIT_BRANCH
echo "env.GIT_BRANCH=${env.GIT_BRANCH},env.GIT_COMMIT=${env.GIT_COMMIT}"
}
setBuildStatus("Test running", "PENDING");
}
}
stage('Test') {
steps {
sh 'CGO_ENABLED=0 GOCACHE=$(pwd)/.cache go test *.go'
}
}
}
post {
success {
setBuildStatus("Test success", "SUCCESS");
}
failure {
setBuildStatus("Test failed", "FAILURE");
}
}
}

Delivery Pipeline#

The delivery pipeline, similar to the test pipeline, first pulls codes in prod branch of the git repo. Then use Docker to build images and push it to remote image repo. (Here we use DockerHub, the withRegistry function takes image repo location and the Credential ID of the repo as parameters). After image been built, the pipeline converts Application YAML file into JSON file, with GIT_COMMIT injected. Finally, the pipeline sends POST requests to KubeVela apiserver (here is http://47.88.24.19/) for creating or updating target application.

Currently, KubeVela apiserver takes JSON object as inputs. Therefore we do extra conversion in the delivery pipeline. In the future, the KubeVela apiserver will further improve and simplify this interaction process. The admission management will be added as well to address the security issue.

In this case we will create an application named cicd-demo-app in Namespace kubevela-demo-namespace. Notice that the Namespace need to be created in Kubernetes in advance. KubeVela apiserver will simplify it in later version.

void setBuildStatus(String message, String state) {
step([
$class: "GitHubCommitStatusSetter",
reposSource: [$class: "ManuallyEnteredRepositorySource", url: "https://github.com/Somefive/KubeVela-demo-CICD-app"],
contextSource: [$class: "ManuallyEnteredCommitContextSource", context: "ci/jenkins/deploy-status"],
errorHandlers: [[$class: "ChangingBuildStatusErrorHandler", result: "UNSTABLE"]],
statusResultSource: [ $class: "ConditionalStatusResultSource", results: [[$class: "AnyBuildResult", message: message, state: state]] ]
]);
}
pipeline {
agent any
stages {
stage('Prepare') {
steps {
script {
def checkout = git branch: 'prod', url: 'https://github.com/Somefive/KubeVela-demo-CICD-app.git'
env.GIT_COMMIT = checkout.GIT_COMMIT
env.GIT_BRANCH = checkout.GIT_BRANCH
echo "env.GIT_BRANCH=${env.GIT_BRANCH},env.GIT_COMMIT=${env.GIT_COMMIT}"
setBuildStatus("Deploy running", "PENDING");
}
}
}
stage('Build') {
steps {
script {
docker.withRegistry("https://registry.hub.docker.com", "DockerHubCredential") {
def customImage = docker.build("somefive/kubevela-demo-cicd-app")
customImage.push()
}
}
}
}
stage('Deploy') {
steps {
sh 'wget -q "https://github.com/mikefarah/yq/releases/download/v4.12.1/yq_linux_amd64"'
sh 'chmod +x yq_linux_amd64'
script {
def app = sh (
script: "./yq_linux_amd64 eval -o=json '.spec' app.yaml | sed -e 's/GIT_COMMIT/$GIT_COMMIT/g'",
returnStdout: true
)
echo "app: ${app}"
def response = httpRequest acceptType: 'APPLICATION_JSON', contentType: 'APPLICATION_JSON', httpMode: 'POST', requestBody: app, url: "http://47.88.24.19/v1/namespaces/kubevela-demo-namespace/applications/cicd-demo-app"
println('Status: '+response.status)
println('Response: '+response.content)
}
}
}
}
post {
success {
setBuildStatus("Deploy success", "SUCCESS");
}
failure {
setBuildStatus("Deploy failed", "FAILURE");
}
}
}

Performance#

After finishing the configuration process described above, the whole process of continuous delivery has already been set up. Let's check how it works.

pipeline-overview

First, we set the VERSION constant in main.go to Bad Version Number, aka,

const VERSION = "Bad Version Number"

Then, we submit this change to dev branch. We can see that the test pipeline in Jenkins is triggered and the failure status is written back to GitHub.

test-pipeline-fail test-github-fail

We edit the VERSION to 0.1.1 again and resubmit it. Now we see that the test pipeline is successfully executed, with the commit in GitHub marked as succeeded.

test-pipeline-success test-github-success

Then we issue a Pull Request to merge dev branch into prod branch.

pull-request

The Jenkins delivery pipeline is triggered once the Pull Request is accepted. After execution finished, the latest commit in prod branch is also marked as succeeded.

deploy-pipeline-success deploy-github-success

$ kubectl get app -n kubevela-demo-namespace
NAME COMPONENT TYPE PHASE HEALTHY STATUS AGE
kubevela-demo-cicd-app kubevela-demo-app-web webservice running true 112s
$ kubectl get deployment -n kubevela-demo-namespace
NAME READY UP-TO-DATE AVAILABLE AGE
kubevela-demo-app-web-v1 2/2 2 2 2m1s

As shown above, the target application is successfully accepted by KubeVela apiserver and related resources are created by KubeVela controller. The current replica number of Deployment is 2. After deleting batchPartition : 0 in the rollout trait of the application, which means confirming current release, the Deployment replica is updated to 5. Now we can access the domain configured in Ingress and get the current version number.

$ kubectl edit app -n kubevela-demo-namespace
application.core.oam.dev/kubevela-demo-cicd-app edited
$ kubectl get deployment -n kubevela-demo-namespace -w
NAME READY UP-TO-DATE AVAILABLE AGE
kubevela-demo-app-web-v1 4/5 5 4 3m39s
kubevela-demo-app-web-v1 5/5 5 5 3m39s
kubevela-demo-app-web-v1 5/5 5 5 3m40s
kubevela-demo-app-web-v1 5/5 5 5 3m40s
$ curl http://kubevela-demo-cicd-app.cf7c0ed25b151437ebe1ef58efc29bca4.us-west-1.alicontainer.com/
Version: 0.1.1

Repeat the steps above. Upgrade the version number to 0.1.2. Finish both test pipeline and delivery pipeline. Then we will see there is a version change to the Deployment managed by the target application. The replica number of the old Deployment decreases from 5 to 3 while the new one contains 2 replica at this moment. If we access the service now, we will find sometimes the old version number is returned and sometimes the new version number is displayed. This is because when rolling update the application, both new version replica and old version replica exist. The incoming traffic will be dispatched to different version replica. Therefore we can observe two different version at the same time.

$ kubectl get deployment -n kubevela-demo-namespace -w
NAME READY UP-TO-DATE AVAILABLE AGE
kubevela-demo-app-web-v1 5/5 5 5 11m
kubevela-demo-app-web-v2 0/0 0 0 0s
kubevela-demo-app-web-v2 0/0 0 0 0s
kubevela-demo-app-web-v2 0/0 0 0 0s
kubevela-demo-app-web-v1 5/5 5 5 12m
kubevela-demo-app-web-v2 0/0 0 0 0s
kubevela-demo-app-web-v2 0/0 0 0 0s
kubevela-demo-app-web-v2 0/0 0 0 0s
kubevela-demo-app-web-v2 0/0 0 0 0s
kubevela-demo-app-web-v2 0/2 0 0 0s
kubevela-demo-app-web-v2 0/2 0 0 0s
kubevela-demo-app-web-v2 0/2 0 0 0s
kubevela-demo-app-web-v2 0/2 2 0 0s
kubevela-demo-app-web-v1 5/5 5 5 12m
kubevela-demo-app-web-v2 1/2 2 1 2s
kubevela-demo-app-web-v2 2/2 2 2 2s
kubevela-demo-app-web-v1 5/3 5 5 13m
kubevela-demo-app-web-v1 5/3 5 5 13m
kubevela-demo-app-web-v1 3/3 3 3 13m
$ curl http://kubevela-demo-cicd-app.cf7c0ed25b151437ebe1ef58efc29bca4.us-west-1.alicontainer.com/
Version: 0.1.1
$ curl http://kubevela-demo-cicd-app.cf7c0ed25b151437ebe1ef58efc29bca4.us-west-1.alicontainer.com/
Version: 0.1.1
$ curl http://kubevela-demo-cicd-app.cf7c0ed25b151437ebe1ef58efc29bca4.us-west-1.alicontainer.com/
Version: 0.1.1
$ curl http://kubevela-demo-cicd-app.cf7c0ed25b151437ebe1ef58efc29bca4.us-west-1.alicontainer.com/
Version: 0.1.1
$ curl http://kubevela-demo-cicd-app.cf7c0ed25b151437ebe1ef58efc29bca4.us-west-1.alicontainer.com/
Version: 0.1.2
$ curl http://kubevela-demo-cicd-app.cf7c0ed25b151437ebe1ef58efc29bca4.us-west-1.alicontainer.com/
Version: 0.1.2
$ curl http://kubevela-demo-cicd-app.cf7c0ed25b151437ebe1ef58efc29bca4.us-west-1.alicontainer.com/
Version: 0.1.2
$ curl http://kubevela-demo-cicd-app.cf7c0ed25b151437ebe1ef58efc29bca4.us-west-1.alicontainer.com/
Version: 0.1.2
$ curl http://kubevela-demo-cicd-app.cf7c0ed25b151437ebe1ef58efc29bca4.us-west-1.alicontainer.com/
Version: 0.1.1

After confirming new services are functioning correctly, we can remove the batchPartition: 0 as described above to complete the whole canary release process.

$ kubectl get deployment -n kubevela-demo-namespace -w
NAME READY UP-TO-DATE AVAILABLE AGE
kubevela-demo-app-web-v1 3/3 3 3 18m
kubevela-demo-app-web-v2 2/2 2 2 5m24s
kubevela-demo-app-web-v2 2/5 2 2 5m36s
kubevela-demo-app-web-v2 2/5 2 2 5m37s
kubevela-demo-app-web-v2 2/5 2 2 5m37s
kubevela-demo-app-web-v2 2/5 5 2 5m37s
kubevela-demo-app-web-v2 3/5 5 3 5m38s
kubevela-demo-app-web-v2 4/5 5 4 5m38s
kubevela-demo-app-web-v2 5/5 5 5 5m39s
kubevela-demo-app-web-v1 3/0 3 3 18m
kubevela-demo-app-web-v1 3/0 3 3 18m
kubevela-demo-app-web-v1 0/0 0 0 18m
kubevela-demo-app-web-v1 0/0 0 0 18m
kubevela-demo-app-web-v2 5/5 5 5 5m41s
kubevela-demo-app-web-v2 5/5 5 5 5m41s
kubevela-demo-app-web-v1 0/0 0 0 18m

Conclusion#

In summary, we executed the whole continuous delivery process successfully. In this process, developers can easily update and deploy their applications, with the help of KubeVela and Jenkins. Besides, developers can use their favourite tools in different stages, such as substituting GitHub with Gitlab, or using TravisCI instead of Jenkins.

Readers might also notice that this progress can not only upgrade the application service, but also change deployment plan via editing app.yaml, such as scaling up or adding sidecars, which works like classical push-style GitOps. About more KubeVela GitOps content, you can refer to other related case studies.

KubeVela Performance Test - Managing Massive Applications

As an application management and integration platform, KubeVela needs to handle thousands of applications in production scenario. To evaluate the performance of KubeVela, develop team has conducted performance tests based on simultated environments and demonstrated the capability of managing a large number of applications concurrently.

Setup#

Cluster Environment#

Working with large clusters requires lots of resources, such as Machines, Network Bandwidth, Storages and many other devices. Therefore, KubeVela team adopts kubemark, an official tool provided by kubernetes, to simulate large clusters by mocking hundreds of kubelets. Each kubelet works like a real node except that they do not run real containers inside pods. The aim of KubeVela performance test mainly focus on whether KubeVela controller can manage thousands of applications effectively, instead of pulling images or executing commands inside pods. As a result, we only need to get resources hosting these fake nodes, also named as hollow-nodes.

We set up the Kubernetes clusters on Alibaba Cloud which includes 5 master nodes and 15 worker nodes. The master nodes wil host Kubernetes core components such as kube-apiserver and kube-controller-manager. The worker nodes need to run other pressure-test related componets, including monitoring tools, KubeVela controller and kubemark pods. Since the major target is to test the performance of KubeVela controller, we do not expect other components to be the bottleneck of the pressure test. To this end, both master nodes and worker nodes are equiped with 32 cores and 128 Gi memory. We use the combination of Prometheus, Loki and Grafana as the monitoring suites and grant them enough resources in avoid of crash caused by Out-of-Memory.

Notice that KubeVela controller and monitoring tools need to be placed on real nodes to function while all pods created during performance tests should be assigned to hollow-nodes correctly. To achieve that, we give different taints to hollow-nodes and real nodes, and add corresponding tolerations to different pods.

Application#

To simulate real applications in production, we design an application template with 2 components and 5 functional traits, including

  • 1 webservice component
    • a scaler trait setting its replica to 3
    • a sidecar trait attaching another container to each pod
    • an ingress trait generating one ingress instance and one service instance
  • 1 worker component
    • a scaler trait also setting its replica to 3
    • a configmap trait generating a new configmap and attaching it to worker pods

In the following experiment, we test the performance of KubeVela controller managing 3,000 Applications (12,000 Pods in total) on 200 nodes. Applications are created in parallel at first, then kept running for a while, and finally deleted from the cluster. Each application will be reconciled multiple times with latencies and consumed resources recorded by monitoring tools.

In practice, we also have another trait used for adding tolerations as described above.

KubeVela Controller#

The KubeVela controller is set up with a group of recommendation configurations as follows

  • Kubernetes Resource
    • 0.5 core CPU
    • 1 Gi Memory
    • 1 replica
  • Program
    • concurrent-reconciles=2 (The number of reconcile threads)
    • kube-api-qps=300 (The qps of kubernetes client used in controller)
    • kube-api-burst=500 (The burst of kubernetes client used in controller)
    • informer-re-sync-interval=20m (The interval of routine reconciles.) We will analyze these settings in the below sections.

To evaluate the performance of KubeVela Controller itself, we disabled the Ingress MutatingWebhook and Application ValidatingWebhook which is beyond the focus of this test but will affect the performance of KubeVela Controller by increasing the latency of creating/patching resources.

Experiments#

Creation#

The creation of all 3,000 applications lasted 25min. Getting all pods running takes a bit longer time, which is out of the scope of KubeVela controller.

For each application creation, it will trigger three turns of reconciling. The usage of CPU will reach 100% in the late period of creation. The memory usage will increase as the number of applications rises. It reaches around 67% at the end of creation.

create-cpu create-memory

The average time of the first turn reconciling is relatively short since it only needs patch finalizer. The second and third turn reconciling contain full reconcile cycles and need more time to process. The following charts record the time consumptions of different period during reconciling applications. The average time is generally below 200ms while 99% of reconciles uses less than 800ms.

create-avg-time create-p99-time

Regular Reconciles#

After creation, applications are reconciled by controller every 20min. the monitoring of 8-hour reconcile process are displayed as below. The CPU usage will come up to 90% once reconcile happens routinely. The memory usage generally keeps a stable pattern, up to 75% memory usage.

med-cpu med-memory

The average reconcile time is under 200ms while 99% are about 800ms~900ms. Each regular reconcile for all applications generally takes around 10min.

med-avg med-p99

Deletion#

The application deletion process is fast and low-resource consumptive. It takes less than 3min to delete all applications. However, notice that the deletion of resources managed by application usually takes longer time. This is because the cleanup of these resources (such as deployments or pods) are not directly controlled by the KubeVela controller. KubeVela controller takes charge of deleting their owner and cleanup them by triggering cascading deletion. In addition, each deletion is associated with two turns of reconcile where the second turn returns immediately when it fails to retrieve target applciation (since it is deleted).

del-cpu del-memory del-avg del-p99

Analysis#

Number of Applications#

The experiment displayed above demonstrates a classical scenario for KubeVela. Although 3,000 applications are successfully managed by the KubeVela controller in this case, it is strongly recommended to adopt a smaller number (such as 2,000) of applications with the above configuration for the following reasons: The time and resource consumption is closely associated with the spec of applications. If a lot of users apply larger applications with more pods and more other resources, 3,000 applications might break the resource limit more easily. The memory and CPU usage shown above is approching resource limits. If memory drains, the KubeVela controller will crash and restart. If high CPU usage maintains for a long time, it might cause a long waiting queue in KubeVela controller which further lead to longer response time for application changes.

Configurations#

There are several parameters users could config to adapt into their own scenario. Using more replica for KubeVela controller do not scale up the ability of KubeVela controller. The leader election mechanism ensures that only one replica will work while others will wait. The aim of multi-replica is to support fast recovery when the working one crash. However, if the crash is caused by OOM, the recovery usually will not be able to fix that. The number of qps and burst in the program configuration should be increased accordingly while scaling up KubeVela controller. These two parameters limit the capability for controller to send requests to apiserver. Generally, in order to scale up KubeVela controller, scale up the resource limits and all the program parameters mentioned above (except the reconcile interval). If you have more applications to manage, add more memory. If you have higher operation frequency, add more CPU and threads, then increase qps and burst accordingly. Longer reconcile interval allows more applications to handle, at the cost of longer time to fix potential underlying resource problems. For example, if one deployment managed by one application disappears, the routine reconciling can discover this problem and fix it.

Scaling Up#

In addition to the experiment described above, we conducted another two experiment to test how well KubeVela controller can scale to larger clusters and more applications.

In a 500-node cluster, we tried to create 5,000 applications with the same spec described above, at the speed of 4 per second and lasted for around 21min. 1 core and 2 Gi memory are granted to KubeVela controller with the use of 4 concurrent reconcile threads. The kube-api-qps and kube-api-burst are raised to 500/800 correspondingly. All 30,000 pods successfully turn into Running phase. The time costs for each reconcile is similar to the previous experiment and CPU/Memory cost is not very high compared to the given resources. The regular reconciles for 5,000 applications takes 7~8 minutes, and no significant resource cost is observed. During this scaling, we found that the throughput of kube-apiserver starts to block the creation of application, as too many resources need to be created while applying applications.

std-cpu std-memory

Scaling up to 12,000 applications on 1,000 nodes is much harder than previous attempts. With the same creation speed, the apiserver will be flooded by lots of pod scheduling requests and finally start to drop application creation request. To overcome this difficulty, we divide the creation process of applications into several stage. Each stage only create 1,000~3,000 applications and the next stage will not begin until all pods are ready. With this strategy, we successfully create 12,000 applications, 24,000 deployments, 12,000 services, 12,000 ingress, 12,000 configmaps and 72,000 pods. The whole process takes about 30 hours. To hold this number of applications, KubeVela controller consumes 1.7 cores and 2.45 Gi memory. It takes about 12min to finish a full turn of regular reconciles for all 12,000 applications.

large-cpu large-memory large-all

Future#

The performance test of KubeVela demonstrates its ability of managing thousands of applications with limited resource consumption. It can also scale up to over 10,000 applications on large clusters with 1,000 nodes. In addition, KubeVela team also conducted similar pressure test for non-deployment based applications such as CloneSet in OpenKruise (which is not enclosed in this report) and reach same conclusions. In the future, we will add more performance tests for more complex scenario like Workflow or MultiCluster.

KubeVela - The Extensible App Platform Based on Open Application Model and Kubernetes

Lei Zhang and Fei Guo

Lei Zhang and Fei Guo

CNCF TOC Member/Kubernetes

7 Dec 2020 12:33pm, by Lei Zhang and Fei Guo

image

Last month at KubeCon+CloudNativeCon 2020, the Open Application Model (OAM) community launched KubeVela, an easy-to-use yet highly extensible application platform based on OAM and Kubernetes.

For developers, KubeVela is an easy-to-use tool that enables you to describe and ship applications to Kubernetes with minimal effort, yet for platform builders, KubeVela serves as a framework that empowers them to create developer-facing yet fully extensible platforms at ease.

The trend of cloud native technology is moving towards pursuing consistent application delivery across clouds and on-premises infrastructures using Kubernetes as the common abstraction layer. Kubernetes, although excellent in abstracting low-level infrastructure details, does introduce extra complexity to application developers, namely understanding the concepts of pods, port exposing, privilege escalation, resource claims, CRD, and so on. We’ve seen the nontrivial learning curve and the lack of developer-facing abstraction have impacted user experiences, slowed down productivity, led to unexpected errors or misconfigurations in production.

Abstracting Kubernetes to serve developers’ requirements is a highly opinionated process, and the resultant abstractions would only make sense had the decision-makers been the platform builders. Unfortunately, the platform builders today face the following dilemma: There is no tool or framework for them to easily extend the abstractions if any.

Thus, many platforms today introduce restricted abstractions and add-on mechanisms despite the extensibility of Kubernetes. This makes easily extending such platforms for developers’ requirements or to wider scenarios almost impossible.

In the end, developers complain those platforms are too rigid and slow in response to feature requests or improvements. The platform builders do want to help but the engineering effort is daunting: any simple API change in the platform could easily become a marathon negotiation around the opinionated abstraction design.

Introducing KubeVela#

With KubeVela, we aim to solve these two challenges in an approach that separates concerns of developers and platform builders.

For developers, KubeVela is an easy-to-use yet extensible tool that enables you to describe and deploy microservices applications with minimal effort. And instead of managing a handful of Kubernetes YAML files, a simple docker-compose style appfile is all you need.

A Sample Appfile#

In this example, we will create a vela.yaml along with your app. This file describes how to build the image, how to deploy the image to Kubernetes, how to access the application and how the system would scale it automatically.

name: testapp
services:
express-server:
image: oamdev/testapp:v1
build:
docker:
file: Dockerfile
contrxt: .
cmd: ["node", "server.js"]
port: 8080
cpu: "0.01"
route:
domain: example.com
rules:
- path: /testapp
rewriteTarget: /
autoscale:
min: 1
max: 4
cpuPercent: 5

Just do: $ vela up, your app will then be alive on https://example.com/testapp.

Behind the Appfile#

The appfile in KubeVela does not have a fixed schema specification, instead, what you can define in this file is determined by what kind of workload types and traits are available in your platform. These two concepts are core concepts from OAM, in detail:

  • Workload type, which declares the characteristics that runtime infrastructure should take into account in application deployment. In the sample above, it defines a “Web Service” workload named express-server as part of your application.
  • Trait, which represents the operation configurations that are attached to an instance of workload type. Traits augment a workload type instance with operational features. In the sample above, it defines a route trait to access the application and an autoscale trait for the CPU based horizontal automatic scaling policy.

Whenever a new workload type or trait is added, it would become immediately available to be declared in the appfile. Let’s say, a new trait named metrics is added, developers could check the schema of this trait by simply $ vela show metrics and define it in the previous sample appfile:

name: testapp
services:
express-server:
type: webservice
image: oamdev/testapp:v1
build:
docker:
file: Dockerfile
contrxt: .
cmd: ["node", "server.js"]
port: 8080
cpu: "0.01"
route:
domain: example.com
rules:
- path: /testapp
rewriteTarget: /
autoscale:
min: 1
max: 4
cpuPercent: 5
metrices:
port: 8080
path: "/metrics"
scheme: "http"
enabled: true

Vela Up#

The vela up command deploys the application defined in appfile to Kubernetes. After deployment, you can use vela status to check how to access your application following the route trait declared in appfile.

Apps deployed with KubeVela will receive a URL (and versioned pre-release URLs) with valid TLS certificate automatically generated via cert-manager. KubeVela also provides a set of commands (i.e. vela logs, vela exec) to best support your application management without becoming a Kubernetes expert. Learn more about vela up and appfile.

KubeVela for Platform Builders#

The above experience cannot be achieved without KubeVela’s innovative offerings to the platform builders as an extensible platform engine. These features are the hidden gems that make KubeVela unique. In details, KubeVela relieves the pains of building developer facing platforms on Kubernetes by doing the following:

  • Application Centric. Behind the appfile, KubeVela enforces “application” as its main API and all KubeVela’s capabilities serve the applications’ requirements only. This is how KubeVela brings application-centric context to the platform by default and changes building such platforms into working around application architecture.
  • Extending Natively. As mentioned in the developer section, an application described by appfile is composed of various pluggable workload types and operation features (i.e. traits). Capabilities from Kubernetes ecosystem can be added to KubeVela as new workload types or traits through Kubernetes CRD registry mechanism at any time.
  • Simple yet Extensible User Interface. Behind the appfile, KubeVela uses CUELang as the “last mile” abstraction engine between user-facing schema and the control plane objects. KubeVela provides a set of built-in abstractions to start with and the platform builders are free to modify them at any time. Capability adding/updating or abstraction changes will all take effect at runtime, neither recompilation nor redeployment of KubeVela is required.

Under the hood, KubeVela core is built on top of Crossplane OAM Kubernetes Runtime with KEDA, Flagger, Prometheus, etc as dependencies, yet its feature pool is “unlimited” and can be extended at any time.

With KubeVela, platform builders now have the tooling support to design and ship any new capabilities with abstractions to end-users with high confidence and low turnaround time. And for a developer, you only need to learn these abstractions, describe the app with them in a single file, and then ship it.

Not Another PaaS System#

Most typical Platform-as-a-Service (PaaS) systems also provide full application management capabilities and aim to improve developer experience and efficiency. In this context, KubeVela shares the same goal.

Though unlike most typical PaaS systems which are either inextensible or create their own addon systems maintained by their own communities. KubeVela is designed to fully leverage the Kubernetes ecosystems as its capability pool. Hence, there’s no additional addon system introduced in this project. For platform builders, a new capability can be installed in KubeVela at any time by simply registering its API resource to OAM and providing a CUE template. We hope and expect that with the help of the open source community, the number of the KubeVela’s capabilities will grow dramatically over time. Learn more about using community capabilities by $vela cap.

So in a nutshell, KubeVela is a Kubernetes plugin for building application-centric abstractions. It leverages the native Kubernetes extensibility and capabilities to resolve a hard problem – making application management enjoyable on Kubernetes.

Learn More#

KubeVela is incubated by the OAM community as the successor of Rudr project, while rather than being a reference implementation, KubeVela intends to be an end-to-end implementation that could be used in wider scenarios. The design of KubeVela’s appfile is also part of the experimental attempt in OAM specification to bring a simplified user experience to developers.

To learn more about KubeVela, please visit KubeVela’s documentation site. The following content are also good next steps:

  • Try out KubeVela following the step-by-step tutorial in its Quick Start page.
  • Give us feedback! KubeVela is still in its early stage and we are happy to ask the community for feedback via OAM Gitter or Slack channel.
  • Extend KubeVela to build your own platforms. If you have an idea for a new workload type, trait or try to build something more complex like a database or AI PaaS with KubeVela, post your idea as a GitHub Issue or propose it to the OAM community, we are eager to know.
  • Contribute to KubeVela. KubeVela is initialized by the open source community with bootstrap contributors from 8+ different organizations. We intend to donate this project to a neutral foundation as soon as it gets stable.