Since Open Application Model invented in 2020, KubeVela has experienced tens of version changes and evolves advanced features towards modern application delivery. Recently, KubeVela has proposed to become a CNCF incubation project and delivered several public talks in the community. As a memorandum, this article will look back into the starting points and give a comprehensive introduction to the state of KubeVela in 2022.

What is KubeVela?

KubeVela is a modern software platform that makes delivering and operating applications across today's hybrid, multi-cloud environments easier, faster and more reliable. It has three main features:

• Infrastructure agnotic: KubeVela is able to deploy your cloud-native application into various destinations, such as Kubernetes multi-clusters, cloud provider runtimes (like Alibaba Cloud, AWS or Azure) and edge devices.
• Programmable: KubeVela has abstraction layers for modeling applications and delivery process. The abstraction layers allow users to use programmable ways to build higher level reusable modules for application delivery and integrate arbitrary third-party projects (like FluxCD, Crossplane, Istio, Prometheus) in the KubeVela system.
• Application-centric: There are rich tools and eco-systems designed around the KubeVela applications, which add extra capabilities for deliverying and operating the applications, including CLI, UI, GitOps, Observability, etc.

KubeVela cares the whole lifecycle of the applications, including both the Day-1 Delivery and the Day-2 Operating stages. It is able to connect with a wide range of Continuous Integration tools, like Jenkins or GitLab CI, and help users deliver and operate applications across hybrid environments.

Why KubeVela?

Challenges and Difficulties​

Nowadays, the fast growing of the cloud native infrastructures has given more and more capabilities for users to deploying applications, such as High Availability and Security, but also exposes an increase number of complexities directly to application developers. For example, the Ingress resource on Kubernetes enables users to expose their applications easily, but developers need to handle the Ingress upgrades when the underlying Kubernetes version shifts, which requires knowledges for the Ingress resource. The hybrid deployment across various cloud provides can make this problem even harder. These difficulties are caused by the lack of operational input in the application definition and developers must face the infrastructure details directly if they want to enjoy the benefits brought by the rich cloud-native community.

Open Application Model​

To tackle the above challenges and bridge the gap between the use of applications and the understanding of infrasturcture details, Open Application Model (OAM) is jointly proposed by Alibaba Cloud and Microsoft Azure in 2020. The aim is to define a consistent application model for application delivery, irrelevant with the platforms and implementations. The defined application model describes an interface for developers on what an application consists of and how it should work. The former one is known as Component in OAM, which is usually used to model the workloads of the application. The latter one is defined as Trait in OAM, which attaches extra capabilities to Components.

KubeVela as OAM​

KubeVela is one of the implementations for the Open Application Model. In KubeVela, the abstraction layer is powered by CUE, a novel configuration programming language which can describe complex rendering logics and work as a superset of JSON. The abstraction layer simplifies the configuration of resources in Kubernetes, which hides the details of implementations and exposes limited parameters to the front developers. With KubeVela application, it is easy for developers to focus on the centric logic of applications, like what container image should be used and how the service should be made accessible. To achieve that, best practices of using Kubernetes native resources are summarized into KubeVela X-Definitions, which provide rendering templates of resources using CUE. These templates can be accessed from various sources, including official repositories, community addons or even self customized implementations by system operators. The templates are mostly infrastructure implemetation agnostic, in other words, not necessarily bond to specific infrastructures. The developers do not need to be aware of the underlying infra when using these templates.

Components & Traits​

The application model divides the abstraction of infra into two different aspects. The Component describes the main workload, which particularly in Kubernetes can be Deployments, StatefulSets, Helm Releases, etc. The Trait on the other hands, describes the added capability for the main workload, such as the scaler trait specifying the number of replicas and the gateway trait aggregates the endpoints for access. The separation of concerns in the design of Component and Trait give high extensibility and reusability to the abstraction. For example, the gateway trait could be backended by different infrastructures like Ingress or HTTPRoute. The application developer who uses the trait only needs to care about the exposed parameters, including the path, port and domain. The trait can be attached to various types of workloads, abstracted by different types of components, such as Deployment, StatefulSet, CloneSet, etc. In the cases where application developers and SRE are in the different teams, KubeVela makes clear division for their responsibilities.

• The platform team providing infrastructures, are responsible to build up X-Definitions where they enforce best practices and deployment confidence.
• The end users only need to choose the Component and Trait provided by the platform team and use them to assemble applications. They can simply enjoy PaaS-like experiences instead of directly interacting with the infra behind.

These are made possible thanks to the flexible, extensible and programmable system of KubeVela and can be applied under varying environments.

Unified Delivery​

Application delivery could happen everywhere. Therefore, another goal for KubeVela application is to build up unified delivery and provide consistent usage for users under various scenarios.

Hybrid-Cloud & Multi-Cluster​

In addition to the abstraction layer, KubeVela also supports hybrid-cloud or multi-cluster architecture natively as modern cloud native applications are not only about containers but involves lots of cloud resources as well. Besides, more and more users and teams start facing the difficulties of deliverying applications to various environments or multi-clusters for different purposes, such as testing or high availability. The KubeVela application allows user to define delivery targets and differentiated configurations through policies. The abstraction helps hide the details of how clusters are registered and connected and provide runtime-agnostic usages to app developers.

Addon Integration​

To enrich the delivery capability, users can leverage KubeVela addons to make extensions to their system. The addons are discoverable, reusable and easy-to-install capability bundles. They usually contain capability providers, including a wide range of third-party projects, like FluxCD, ClickHouse, Crossplane, etc. Addons not only install those projects into the system but create corresponding definitions for the integration concurrently, which extends the types of Component and Trait that application developers are able to use. The KubeVela community currently have 50+ addons already. Platform builders could enjoy these out-of-box integrations in systems depending on their customized demands.

With addons enabled in the system, it would be possible for end users to assemble applications in more customized ways, such as deploying cloud resources or using advanced workloads.

KubeVela Workflow​

While the Open Application Model defines the composition of an application, in real cases, the delivery process of the compositions could still vary a lot. For example, the different components in one application could have inter dependencies or data passing where delivery steps must be executed in specific order. Furthermore, the delivery process sometimes also involves more actions apart from the delivery of resources, such as rollouts or notifications. An extensible workflow is therefore designed to fulfill the needs of the process customization in KubeVela. Similar to Component and Trait, KubeVela workflow also leverages CUE to define workflow steps, providing flexibility, extensibility and programmability. It can be seen as another form of Infrastructure as Code (IaC). A bunch of build-in workflow steps has already provided rich out-of-box capabilities in KubeVela, such as making multi-cluster deployments and sending notifications through slack or email. The lightweight engine ensures the high performance and safety of step executions, compared to other types of engines involving running extra containers. Differ from the Component and Trait definitions in KubeVela, the WorkflowStep definition does not render templates into resources. Instead, it describes the actions to be executed in the step, which calls underlying atomic functions in various providers. With the use of workflow and addons, users are able to build arbitrary delivery process and make customized integrations. For example, it is possible to let the Continuous Integration tools to trigger the delivery of KubeVela applications and implement the GitOps solutions combining FluxCD and other addons.

Day-2 Management​

KubeVela cares more other than Day-1 Delivery. It also provides a unified Day-2 application management capability for all it's extensibility. The day-2 management is necessary for system operators and application developers to make continuous operation for the delivered applications and ensure the applications are always under control.

Resource Management​

The basic capabilities for application management are for its resources. KubeVela's core controller continuously watches the difference between the current state and the desired state of delivered resources. It makes sure that the live spec is accord with the declared spec recorded in the delivery process and therefore effectively prevents any configuration drifts. Besides, the automated garbage collection help recycle the resources that are not in-use during upgrades or deletion. There are also times resources need to be shared across multiple applications. These are all made possible in KubeVela application through the use of policies.

Version Control​

KubeVela application keeps history records for deliveries. These snapshots are useful when new version publish are out of expectations. The change inspectation could be used to diagnose the possible error changes and the rollback allows fast recovery to the previous successful states.

Observability​

KubeVela treats observability as first class citizen. It is the eyes to users for monitoring the state of applications and observing exceptions. There are multiple tools and methods in KubeVela to do the observation job. One of the most straightforward way is to use the CLI tool of KubeVela. The Vela CLI is able to provide in-time status info for the application in fine-grain or aggregated level. For users that prefer web interfaces, VelaUX provides an alternative way to view application status. In the cases applications are monitored through third-party projects, such as Grafana, Prometheus or Loki, KubeVela further provides addons for bootstrapping the observability infrastructures and empower users to customize the observing rules as codes in applications, through the abstraction layer. A series of out-of-box metrics and dashboards give users the basic capability of automated system observability. These can be used to diagnose system level exceptions and help improve the overall performance.

Eco-system​

In addition to the above mentioned tools, KubeVela also has several other tools in the eco-systems to facilitate application delivery.

• Vela CLI: KubeVela CLI provides various commands that helps you to operate applications, such as managing definitions, viewing resources, restarting workflow, rolling versions.
• VelaUX: VelaUX is the Web UI for KubeVela. Besides, it incorporates business logics into fundamental APIs and provides out-of-box user experiences for non-k8s-expert users.
• Terraform Controller: The terraform controller in KubeVela allows users to use Terraform to manage cloud resources through Kubernetes Custom Resources.
• Cluster Gateway: The gateway that provides unified multi-cluster access interface. Working as Kubernetes Aggregated API Server, the gateway leverages the native Authentication and Authorization modules and enforces secure and transparent access to managed clusters.
• VelaD: Building on top of k3s & k3d, VelaD integrates KubeVela with Kubernetes cores, which can be extremely helpful for building dev/test environment.
• Vela Prism: The extension API server for KubeVela built upon the Kubernetes Aggregated API Server. It projects native APIs like creating dashboards on Grafana into Kubernetes resource APIs, so that users can manage 3rd-party resources as Kubernetes native resources.
• Vela Workflow: The workflow engine translates CUE-based steps and executes them. It works as a pure delivery tool and can be used aside by the KubeVela application. Compared to Tekton, it mainly organize the process in CUE style, instead of using Pods and Jobs directly.

Stability​

To ensure KubeVela is able to handle certain amount of applications under limited resources, multiple load testings have been conducted under various circumstances. The experiments have demonstrated that the performance of KubeVela system is capable of dealing thousands of applications in an ordinary-sized cluster. The observability infrastructure further exposes the bottleneck of KubeVela and guides system operators to do customized tunning to improve the performance in specific use environments.

In a nutshell

Currently, KubeVela has already been applied in production by a number of adopters from various areas. Some mainly use KubeVela's abstraction capability to simplify the use and deploy of applications. Some build application-centric management system upon KubeVela. Some use the customized workflow to orchestrate the delivery process. It is especially welcomed in high-tech industries and shown to be helpful for delivering and managing enourmous applications.

The KubeVela community has attracted world-wide contributors and continuously evolves over the past two years. Nowadays, there are over 200 contributors from various contries have participated in the developing of KubeVela. Thousands of issues have been raised and 85% of them are already solved. There are also bi-weekly community meetings held in both English and Chinese community.

With more and more people coming into the community, KubeVela is consistently upgrading itself to fit into more complex, varying use cases and scenarios.

Application Delivery on Kubernetes​

The cloud-native landscape is formed by a fast-growing ecosystem of tools with the aim of improving the development of modern applications in a cloud environment. Kubernetes has become the de facto standard to deploy enterprise workloads by improving development speed, and accommodating the needs of a dynamic environment.

Kubernetes offers a comprehensive set of entities that enables any potential application to be deployed into it, independent of its complexity. This however has a significant impact from the point of view of its adoption. Kubernetes is becoming as complex as it is powerful, and that translates into a steep learning curve for newcomers into the ecosystem. Thus, this has generated a new trend focused on providing developers with tools that improve their day-to-day activities without losing the underlying capabilities of the underlying system.

Napptive Application Platform​

The NAPPTIVE Playground offers a cloud native application platform focused on providing a simplified method to operate on Kubernetes clusters and deploy complex applications without needing to work with low-level Kubernetes entities. This is especially important to be able to accommodate non-developer users' personas as the computing resources of a company are being consolidated in multi-purpose, multi-tenant clusters. Data scientists, business analysts, modeling experts and many more can benefit from simple solutions that do not require any type of knowledge of cloud computing infrastructures, or orchestration systems such as Kubernetes, which enable them to run their applications with ease in the existing infrastructure.

Any tool that works in this space must start by analyzing the existing abstractions. In particular, at Napptive, we focus on the Applications. This quite understood abstraction is not present in Kubernetes, requiring users to manually tag, identify and reason about the different components involved in an application. The Open Application Model provides an excellent abstraction to represent any type of application independently of the cloud provider, containerization technology, or deployment framework. The model is highly customizable by means of adding Traits, Policies, or new Component Definitions.

KubeVela in Napptive​

The Napptive Playground leverages Kubevela as the OAM runtime for Kubernetes deployments. Our Playground provides an environment abstraction with multi-tenant guarantees that is equivalent to partitioning a shared cluster by means of differentiated namespaces. The benefit of our approach is the transparency of its configuration using a higher abstraction level that does not involve any Kubernetes knowledge.

The following diagram describes the overall architecture of Napptive and Kubevela deployed in a Kubernetes cluster.

The user has the ability to interact with the cluster by using the Napptive user interface (CLI or Web UI), or by means of using the standard Kubernetes API with kubectl. Isolated environments can be easily created to establish the logical separations such as type of environment (e.g., deployment, staging, production), or purpose (e.g., projectA, projectB), or any other approach to differentiate where to deploy an application. Once the OAM application is deployed on Kubernetes, Kubevela is in charge of managing the low-level application lifecycle creating the appropriate Kubernetes entities as a result. One of the main differences with other adopters of Kubevela, is the fact that we use Kubevela in a multi-tenant environment. The following figure shows the specifics of an application deployment in this type of scenario.

The Napptive Playground is integrated with Kubernetes RBAC and offers a native user management layer that works in both on-premise and cloud deployments. User identity is associated with each environment, and Kubevela is able to take that information to ensure that users can only access their allowed resources. Once a user deploys an OAM application, Kubevela will intercept the call and attach the user identity as an annotation. After that, the rendering process will ensure that the user has access to the entities (e.g., trait, component, policy, workflow step definitions) and that the target namespaces are also accessible by the user. Once the application render is ready, the workflow orchestrator will create the different entities in the Kubernetes namespace.

Napptive in the Community​

In terms of our involvement with the OAM/Kubevela community, it has evolved overtime from being passive members of the community simply exploring the possibilities of OAM in Kubernetes, to becoming active contributor members in different areas. In particular, we have closely worked with the core Kubevela development team to test and overcome the different challenges related to using a multi-tenant, RBAC compliance installation of Kubevela. Security in this type of installation is critical and it is one of our main focuses within the Kubevela community. We are committed to continue working with the community not only to ensure that multi-tenancy is maintained throughout the different releases, but also to add our own perspective representing our customer use cases.

The Kubevela community is growing at a fast pace, and we try to contribute our adaptations, features, feedback, and thoughts back to the community. This type of framework is deployed in a multitude of environments. The most common one is probably where one or more clusters are inside a company. In our particular application, we are interested in exploring methods to offer computing capabilities for a variety of user personas in a shared cluster, and we contribute our view and experience in the community.

Future Evolution​

In terms of future evolution, we believe the Napptive Playground is a great tool to experience working with the Open Application Model without the need of installing new clusters and frameworks. From the point of view of our contributions in the community, we are internally working in exploring QA mechanisms that ensure that customer applications remain working after upgrades, and identifying potential incompatibilities between releases. Once that work is ready, we plan to contribute our testing environment setup back to the community so that it can be adopted in the main branch. We are also excited with new functionalities that have been recently added to Kubevela such as multi-cluster support, and are exploring methods to adopt it inside the Playground. Moreover, we are actively working on an OAM-compatible application catalog that will simplify the way organizations store and make accessible application definitions so that they can be deployed into a cluster from a central repository. The catalog focuses on the OAM entities and relies on existing container registries for storing the image.

The community has released the new milestone release v1.6 of KubeVela during the 2022 Apsara Conference. This release is a qualitative change in KubeVela from application delivery to application management. It also creates a precedent in the industry to build an application platform with delivery and management integrated based on a extensible model.

KubeVela 1.5 was released recently. This release brings more convenient application delivery capabilities to the community, including system observability, CloudShell terminals that move the Vela CLI to the browser, enhanced canary releases, and optimized multi-environment application delivery workflows. It also improved KubeVela's high extensibility as an application delivery platform. The community has started to promote the project to the CNCF Incubation stage. It has absorbed the practice sharing of multiple benchmark users in many community meetings, which proves the community's healthy development. The project is now mature to some extent, and its adoption has made periodical achievements, thanks to the contributions of more than 200 developers in the community.

You may have learned from this blog that we can use vela to manage cloud resources (like s3 bucket, AWS EIP and so on) via the terraform plugin. We can create an application which contains some cloud resource components and this application will generate these cloud resources, then we can use vela to manage them.

Sometimes we already have some Terraform cloud resources which may be created and managed by the Terraform binary or something else. In order to have the benefits of using KubeVela to manage the cloud resources or just maintain consistency in the way you manage cloud resources, we may want to import these existing Terraform cloud resources into KubeVela and use vela to manage them. But if we just create an application which describes these cloud resources, the cloud resources will be recreated and may lead to errors. To fix this problem, we made a simple backup_restore tool. This blog will show you how to use the backup_restore tool to import your existing Terraform cloud resources into KubeVela.

If you're looking for something to glue Terraform ecosystem with the Kubernetes world, congratulations! You're getting exactly what you want in this blog.

We will introduce how to integrate terraform modules into KubeVela by fixing a real world problem -- "Fixing the Developer Experience of Kubernetes Port Forwarding" inspired by article from Alex Ellis.

In general, this article will be divided into two parts:

• Part.1 will introduce how to glue Terraform with KubeVela, it needs some basic knowledge of both Terraform and KubeVela. You can just skip this part if you don't want to extend KubeVela as a Developer.
• Part.2 will introduce how KubeVela can 1) provision a Cloud ECS instance by KubeVela with public IP; 2) Use the ECS instance as a tunnel sever to provide public access for any container service within an intranet environment.

OK, let's go!

KubeVela is a modern software delivery control panel. The goal is to make application deployment and O&M simpler, more agile, and more reliable in today's hybrid multi-cloud environment. Since the release of Version 1.1, the KubeVela architecture has naturally solved the delivery problems of enterprises in the hybrid multi-cloud environments and has provided sufficient scalability based on the OAM model, which makes it win the favor of many enterprise developers. This also accelerates the iteration of KubeVela.

In Version 1.2, we released an out-of-the-box visual console, which allows the end user to publish and manage diverse workloads through the interface. The release of Version 1.3 improved the expansion system with the OAM model as the core and provides rich plug-in functions. It also provides users with a large number of enterprise-level functions, including LDAP permission authentication, and provides more convenience for enterprise integration. You can obtain more than 30 addons in the addons registry of the KubeVela community. There are well-known CNCF projects (such as argocd, istio, and traefik), database middleware (such as Flink and MySQL), and hundreds of cloud vendor resources.

In Version 1.4, we focused on making application delivery safe, foolproof, and transparent. We added core functions, including multi-cluster permission authentication and authorization, a complex resource topology display, and a one-click installation control panel. We comprehensively strengthened the delivery security in multi-tenancy scenarios, improved the consistent experience of application development and delivery, and made the application delivery process more transparent.