Helm Chart Delivery in Multi-Cluster

July 7, 2022 · 8 min read

KubeVela Team

Helm Charts are very popular that you can find almost 10K different software packaged in this way. While in today's multi-cluster/hybrid cloud business environment, we often encounter these typical requirements: distribute to multiple specific clusters, specific group distributions according to business need, and differentiated configurations for multi-clusters.

In this blog, we'll introduce how to use KubeVela to do multi cluster delivery for Helm Charts.

If you don't have multi clusters, don't worry, we'll introduce from scratch with only Docker or Linux System required. You can also refer to the basic helm chart delivery in single cluster.

Prerequisites

Docker v20.10.5+ (runc >= v1.0.0-rc93), or Linux system
VelaD, a lightweight deployment tool to set up KubeVela with Kubernetes.

Prepare Clusters

This section is preparation for multi-cluster, we will start from scratch for convenience. if you're already KubeVela users and have multi-clusters joined, you can skip this section.

Install KubeVela control plane

velad install

Export the KubeConfig for the newly created cluster

export KUBECONFIG=$(velad kubeconfig --name default --host)

Now you have successfully installed KubeVela. You can join your cluster to kubevela by:

vela cluster join <path-to-kubeconfig-of-cluster> --name foo

VelaD can also provide K3s clusters for convenience.

Create and Join a cluster created by velad named foo

velad install --name foo --cluster-only
vela cluster join $(velad kubeconfig --name foo --internal) --name foo

As a fully extensible control plane, most of KubeVela's capabilities are pluggable. The following steps will guide you to install some addons for different capabilities.

Enable velaux addon, it will provide UI console for KubeVela

vela addon enable velaux

Enable fluxcd addon for helm component delivery

vela addon enable fluxcd

If you have already enabled the fluxcd addon before you joined the new cluster, you NEED to enable the addon for the newly joined cluster by:

vela addon enable fluxcd --clusters foo

Finally, we have finished all preparation, you can check the clusters joined:

$ vela cluster ls
CLUSTER ALIAS   TYPE            ENDPOINT                ACCEPTED    LABELS
local           Internal        -                       true
foo             X509Certificate https://172.20.0.6:6443 true

One cluster named local is the KubeVela control plane, another one named foo is the cluster we just joined.

Deploy across multi clusters

We can use topology policy to specify the delivery topology for helm chart like the following command:

cat <<EOF | vela up -f -
apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: helm-hello
spec:
  components:
    - name: hello
      type: helm
      properties:
        repoType: "helm"
        url: "https://jhidalgo3.github.io/helm-charts/"
        chart: "hello-kubernetes-chart"
        version: "3.0.0"
  policies:
    - name: foo-cluster-only
      type: topology
      properties:
        clusters: ["foo"]
EOF

The clusters field of topology policy is a slice, you can specify multiple cluster names here. You can also use label selector or specify namespace with that, refer to the reference docs for more details.

After deployed, you can check the deployed application by:

vela status helm-hello

The expected output should be as follows if deployed successfully:

About:

  Name:         helm-hello
  Namespace:    default
  Created at:   2022-06-09 19:14:57 +0800 CST
  Status:       running

Workflow:

  mode: DAG
  finished: true
  Suspend: false
  Terminated: false
  Steps
  - id:vtahj5zrz4
    name:deploy-foo-cluster-only
    type:deploy
    phase:succeeded
    message:

Services:

  - Name: hello
    Cluster: foo  Namespace: default
    Type: helm
    Healthy Fetch repository successfully, Create helm release successfully
    No trait applied

You can check the deployed resource by:

$ vela status helm-hello --tree
CLUSTER       NAMESPACE     RESOURCE             STATUS
foo       ─── default   ─┬─ HelmRelease/hello    updated
                         └─ HelmRepository/hello updated

You can also check the deployed resource by VelaUX.

Check Resources from UI console

By using the velaux UI console, you can get even more information with a unified experience for multi clusters. You can refer to this doc to learn how to visit VelaUX.

With the help of UI, you can:

Check pod status and event from different clusters:

resource-detail

Check pod logs from different clusters:

resource-detail

Check resource topology and status:

resource-detail

Deploy with override configurations

In some cases, we will deploy helm chart into different clusters with different values, then we can use the override policy.

Below is a complex example that we will deploy one helm chart into two clusters and specify different values for each cluster. Let's deploy it:

cat <<EOF | vela up -f -
apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: helm-hello
spec:
  components:
    - name: hello
      type: helm
      properties:
        repoType: "helm"
        url: "https://jhidalgo3.github.io/helm-charts/"
        chart: "hello-kubernetes-chart"
        version: "3.0.0"
  policies:
    - name: topology-local
      type: topology
      properties:
        clusters: ["local"]
    - name: topology-foo
      type: topology
      properties:
        clusters: ["foo"]
    - name: override-local
      type: override
      properties:
        components:
          - name: hello
            properties:
              values:
                configs:
                  MESSAGE: Welcome to Control Plane Cluster!
    - name: override-foo
      type: override
      properties:
        components:
          - name: hello
            properties:
              values:
                configs:
                  MESSAGE: Welcome to Your New Foo Cluster!
  workflow:
    steps:
      - name: deploy2local
        type: deploy
        properties:
          policies: ["topology-local", "override-local"]
      - name: manual-approval
        type: suspend
      - name: deploy2foo
        type: deploy
        properties:
          policies: ["topology-foo", "override-foo"]
EOF

Note: If you feel the policy and workflow is a bit complex, you can make them as an external object and just reference the object, the usage is the same with the container delivery.

The deploy process has three steps:

1) deploy to local cluster;
2) wait for manual approval;
3) deploy to foo cluster.

So you will find it was suspended after the first step, just like follows:

$ vela status helm-hello
About:

  Name:         helm-hello
  Namespace:    default
  Created at:   2022-06-09 19:38:13 +0800 CST
  Status:       workflowSuspending

Workflow:

  mode: StepByStep
  finished: false
  Suspend: true
  Terminated: false
  Steps
  - id:ww4cydlvee
    name:deploy2local
    type:deploy
    phase:succeeded
    message:
  - id:xj6hu97e1e
    name:manual-approval
    type:suspend
    phase:succeeded
    message:

Services:

  - Name: hello
    Cluster: local  Namespace: default
    Type: helm
    Healthy Fetch repository successfully, Create helm release successfully
    No trait applied

You can check the helm chart deployed in control plane with the value "Welcome to Control Plane Cluster!".

vela port-forward helm-hello

It will automatically prompt with your browser with the following page:

resource-detail

Let's continue the workflow as we have checked the deployment has succeeded.

vela workflow resume helm-hello

Then it will deploy to the foo cluster, you can check the resources with detailed information:

$ vela status helm-hello --tree --detail
CLUSTER       NAMESPACE     RESOURCE             STATUS    APPLY_TIME          DETAIL
foo       ─── default   ─┬─ HelmRelease/hello    updated   2022-06-09 19:38:13 Ready: True  Status: Release reconciliation succeeded  Age: 64s
                         └─ HelmRepository/hello updated   2022-06-09 19:38:13 URL: https://jhidalgo3.github.io/helm-charts/  Age: 64s  Ready: True
                                                                               Status: stored artifact for revision 'ab876069f02d779cb4b63587af1266464818ba3790c0ccd50337e3cdead44803'
local     ─── default   ─┬─ HelmRelease/hello    updated   2022-06-09 19:38:13 Ready: True  Status: Release reconciliation succeeded  Age: 7m34s
                         └─ HelmRepository/hello updated   2022-06-09 19:38:13 URL: https://jhidalgo3.github.io/helm-charts/  Age: 7m34s  Ready: True
                                                                               Status: stored artifact for revision 'ab876069f02d779cb4b63587af1266464818ba3790c0ccd50337e3cdead44803'

Use port forward again:

vela port-forward helm-hello

Then it will prompt some selections:

? You have 2 deployed resources in your app. Please choose one:  [Use arrows to move, type to filter]
> Cluster: foo | Namespace: default | Kind: HelmRelease | Name: hello
  Cluster: local | Namespace: default | Kind: HelmRelease | Name: hello

Choose the option with cluster foo, then you'll see the result that has was overridden with new message.

$ curl http://127.0.0.1:8080/
...snip...
      <div id="message">
  Welcome to Your New Foo Cluster!
</div>
...snip...

Specify different value file for different environment

You can choose different value file present in a helm chart for different environment. eg:

Please make sure your local cluster have two namespaces "test" and "prod" which represent two environments in our example.

We use the chart hello-kubernetes-chart as an example.This chart has two values files. You can pull this chart and have a look all contains files in it:

$ tree ./hello-kubernetes-chart
./hello-kubernetes-chart
├── Chart.yaml
├── templates
│ ├── NOTES.txt
│ ├── _helpers.tpl
│ ├── config-map.yaml
│ ├── deployment.yaml
│ ├── hpa.yaml
│ ├── ingress.yaml
│ ├── service.yaml
│ ├── serviceaccount.yaml
│ └── tests
│ └── test-connection.yaml
├── values-production.yaml
└── values.yaml

As we can see, there are values files values.yaml values-production.yaml in this chart.

cat <<EOF | vela up -f -
apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: hello-kubernetes
spec:
  components:
    - name: hello-kubernetes
      type: helm
      properties:
        repoType: "helm"
        url: "https://wangyikewxgm.github.io/my-charts/"
        chart: "hello-kubernetes-chart"
        version: "0.1.0"

  policies:
    - name: topology-test
      type: topology
      properties:
        clusters: ["local"]
        namespace: "test"
    - name: topology-prod
      type: topology
      properties:
        clusters: ["local"]
        namespace: "prod"
    - name: override-prod
      type: override
      properties:
        components:
          - name: hello-kubernetes
            properties:
              valuesFiles:
                - "values-production.yaml"
  workflow:
    steps:
      - name: deploy2test
        type: deploy
        properties:
          policies: ["topology-test"]
      - name: deploy2prod
        type: deploy
        properties:
          policies: ["topology-prod", "override-prod"]  
EOF

Access the endpoints of application:

vela port-forward hello-kubernetes

If you choose Cluster: local | Namespace: test | Kind: HelmRelease | Name: hello-kubernetes you will see:

If you choose Cluster: local | Namespace: prod | Kind: HelmRelease | Name: hello-kubernetes you will see:

Clean up

If you're using velad for this demo, you can clean up very easily by:

Clean up the foo cluster

velad uninstall -n foo

Clean up the default cluster

velad uninstall

What's More?

With the help of KubeVela and its addon, you can get the capability of Canary Rollout for your helm charts!

Go and ship Helm chart with KubeVela, makes deploying and operating applications across today's hybrid, multi-cloud environments easier, faster and more reliable.

Prerequisites​

Prepare Clusters​

Deploy across multi clusters​

Check Resources from UI console​

Deploy with override configurations​

Specify different value file for different environment​

Clean up​

What's More?​

Prerequisites

Prepare Clusters

Deploy across multi clusters

Check Resources from UI console

Deploy with override configurations

Specify different value file for different environment

Clean up

What's More?