Since Open Application Model invented in 2020, KubeVela has experienced tens of version changes and evolves advanced features towards modern application delivery. Recently, KubeVela has proposed to become a CNCF incubation project and delivered several public talks in the community. As a memorandum, this article will look back into the starting points and give a comprehensive introduction to the state of KubeVela in 2022.

What is KubeVela?

KubeVela is a modern software platform that makes delivering and operating applications across today's hybrid, multi-cloud environments easier, faster and more reliable. It has three main features:

• Infrastructure agnotic: KubeVela is able to deploy your cloud-native application into various destinations, such as Kubernetes multi-clusters, cloud provider runtimes (like Alibaba Cloud, AWS or Azure) and edge devices.
• Programmable: KubeVela has abstraction layers for modeling applications and delivery process. The abstraction layers allow users to use programmable ways to build higher level reusable modules for application delivery and integrate arbitrary third-party projects (like FluxCD, Crossplane, Istio, Prometheus) in the KubeVela system.
• Application-centric: There are rich tools and eco-systems designed around the KubeVela applications, which add extra capabilities for deliverying and operating the applications, including CLI, UI, GitOps, Observability, etc.

KubeVela cares the whole lifecycle of the applications, including both the Day-1 Delivery and the Day-2 Operating stages. It is able to connect with a wide range of Continuous Integration tools, like Jenkins or GitLab CI, and help users deliver and operate applications across hybrid environments.

Why KubeVela?

Challenges and Difficulties​

Nowadays, the fast growing of the cloud native infrastructures has given more and more capabilities for users to deploying applications, such as High Availability and Security, but also exposes an increase number of complexities directly to application developers. For example, the Ingress resource on Kubernetes enables users to expose their applications easily, but developers need to handle the Ingress upgrades when the underlying Kubernetes version shifts, which requires knowledges for the Ingress resource. The hybrid deployment across various cloud provides can make this problem even harder. These difficulties are caused by the lack of operational input in the application definition and developers must face the infrastructure details directly if they want to enjoy the benefits brought by the rich cloud-native community.

Open Application Model​

To tackle the above challenges and bridge the gap between the use of applications and the understanding of infrasturcture details, Open Application Model (OAM) is jointly proposed by Alibaba Cloud and Microsoft Azure in 2020. The aim is to define a consistent application model for application delivery, irrelevant with the platforms and implementations. The defined application model describes an interface for developers on what an application consists of and how it should work. The former one is known as Component in OAM, which is usually used to model the workloads of the application. The latter one is defined as Trait in OAM, which attaches extra capabilities to Components.

KubeVela as OAM​

KubeVela is one of the implementations for the Open Application Model. In KubeVela, the abstraction layer is powered by CUE, a novel configuration programming language which can describe complex rendering logics and work as a superset of JSON. The abstraction layer simplifies the configuration of resources in Kubernetes, which hides the details of implementations and exposes limited parameters to the front developers. With KubeVela application, it is easy for developers to focus on the centric logic of applications, like what container image should be used and how the service should be made accessible. To achieve that, best practices of using Kubernetes native resources are summarized into KubeVela X-Definitions, which provide rendering templates of resources using CUE. These templates can be accessed from various sources, including official repositories, community addons or even self customized implementations by system operators. The templates are mostly infrastructure implemetation agnostic, in other words, not necessarily bond to specific infrastructures. The developers do not need to be aware of the underlying infra when using these templates.

Components & Traits​

The application model divides the abstraction of infra into two different aspects. The Component describes the main workload, which particularly in Kubernetes can be Deployments, StatefulSets, Helm Releases, etc. The Trait on the other hands, describes the added capability for the main workload, such as the scaler trait specifying the number of replicas and the gateway trait aggregates the endpoints for access. The separation of concerns in the design of Component and Trait give high extensibility and reusability to the abstraction. For example, the gateway trait could be backended by different infrastructures like Ingress or HTTPRoute. The application developer who uses the trait only needs to care about the exposed parameters, including the path, port and domain. The trait can be attached to various types of workloads, abstracted by different types of components, such as Deployment, StatefulSet, CloneSet, etc. In the cases where application developers and SRE are in the different teams, KubeVela makes clear division for their responsibilities.

• The platform team providing infrastructures, are responsible to build up X-Definitions where they enforce best practices and deployment confidence.
• The end users only need to choose the Component and Trait provided by the platform team and use them to assemble applications. They can simply enjoy PaaS-like experiences instead of directly interacting with the infra behind.

These are made possible thanks to the flexible, extensible and programmable system of KubeVela and can be applied under varying environments.

Unified Delivery​

Application delivery could happen everywhere. Therefore, another goal for KubeVela application is to build up unified delivery and provide consistent usage for users under various scenarios.

Hybrid-Cloud & Multi-Cluster​

In addition to the abstraction layer, KubeVela also supports hybrid-cloud or multi-cluster architecture natively as modern cloud native applications are not only about containers but involves lots of cloud resources as well. Besides, more and more users and teams start facing the difficulties of deliverying applications to various environments or multi-clusters for different purposes, such as testing or high availability. The KubeVela application allows user to define delivery targets and differentiated configurations through policies. The abstraction helps hide the details of how clusters are registered and connected and provide runtime-agnostic usages to app developers.

Addon Integration​

To enrich the delivery capability, users can leverage KubeVela addons to make extensions to their system. The addons are discoverable, reusable and easy-to-install capability bundles. They usually contain capability providers, including a wide range of third-party projects, like FluxCD, ClickHouse, Crossplane, etc. Addons not only install those projects into the system but create corresponding definitions for the integration concurrently, which extends the types of Component and Trait that application developers are able to use. The KubeVela community currently have 50+ addons already. Platform builders could enjoy these out-of-box integrations in systems depending on their customized demands.

With addons enabled in the system, it would be possible for end users to assemble applications in more customized ways, such as deploying cloud resources or using advanced workloads.

KubeVela Workflow​

While the Open Application Model defines the composition of an application, in real cases, the delivery process of the compositions could still vary a lot. For example, the different components in one application could have inter dependencies or data passing where delivery steps must be executed in specific order. Furthermore, the delivery process sometimes also involves more actions apart from the delivery of resources, such as rollouts or notifications. An extensible workflow is therefore designed to fulfill the needs of the process customization in KubeVela. Similar to Component and Trait, KubeVela workflow also leverages CUE to define workflow steps, providing flexibility, extensibility and programmability. It can be seen as another form of Infrastructure as Code (IaC). A bunch of build-in workflow steps has already provided rich out-of-box capabilities in KubeVela, such as making multi-cluster deployments and sending notifications through slack or email. The lightweight engine ensures the high performance and safety of step executions, compared to other types of engines involving running extra containers. Differ from the Component and Trait definitions in KubeVela, the WorkflowStep definition does not render templates into resources. Instead, it describes the actions to be executed in the step, which calls underlying atomic functions in various providers. With the use of workflow and addons, users are able to build arbitrary delivery process and make customized integrations. For example, it is possible to let the Continuous Integration tools to trigger the delivery of KubeVela applications and implement the GitOps solutions combining FluxCD and other addons.

Day-2 Management​

KubeVela cares more other than Day-1 Delivery. It also provides a unified Day-2 application management capability for all it's extensibility. The day-2 management is necessary for system operators and application developers to make continuous operation for the delivered applications and ensure the applications are always under control.

Resource Management​

The basic capabilities for application management are for its resources. KubeVela's core controller continuously watches the difference between the current state and the desired state of delivered resources. It makes sure that the live spec is accord with the declared spec recorded in the delivery process and therefore effectively prevents any configuration drifts. Besides, the automated garbage collection help recycle the resources that are not in-use during upgrades or deletion. There are also times resources need to be shared across multiple applications. These are all made possible in KubeVela application through the use of policies.

Version Control​

KubeVela application keeps history records for deliveries. These snapshots are useful when new version publish are out of expectations. The change inspectation could be used to diagnose the possible error changes and the rollback allows fast recovery to the previous successful states.

Observability​

KubeVela treats observability as first class citizen. It is the eyes to users for monitoring the state of applications and observing exceptions. There are multiple tools and methods in KubeVela to do the observation job. One of the most straightforward way is to use the CLI tool of KubeVela. The Vela CLI is able to provide in-time status info for the application in fine-grain or aggregated level. For users that prefer web interfaces, VelaUX provides an alternative way to view application status. In the cases applications are monitored through third-party projects, such as Grafana, Prometheus or Loki, KubeVela further provides addons for bootstrapping the observability infrastructures and empower users to customize the observing rules as codes in applications, through the abstraction layer. A series of out-of-box metrics and dashboards give users the basic capability of automated system observability. These can be used to diagnose system level exceptions and help improve the overall performance.

Eco-system​

In addition to the above mentioned tools, KubeVela also has several other tools in the eco-systems to facilitate application delivery.

• Vela CLI: KubeVela CLI provides various commands that helps you to operate applications, such as managing definitions, viewing resources, restarting workflow, rolling versions.
• VelaUX: VelaUX is the Web UI for KubeVela. Besides, it incorporates business logics into fundamental APIs and provides out-of-box user experiences for non-k8s-expert users.
• Terraform Controller: The terraform controller in KubeVela allows users to use Terraform to manage cloud resources through Kubernetes Custom Resources.
• Cluster Gateway: The gateway that provides unified multi-cluster access interface. Working as Kubernetes Aggregated API Server, the gateway leverages the native Authentication and Authorization modules and enforces secure and transparent access to managed clusters.
• VelaD: Building on top of k3s & k3d, VelaD integrates KubeVela with Kubernetes cores, which can be extremely helpful for building dev/test environment.
• Vela Prism: The extension API server for KubeVela built upon the Kubernetes Aggregated API Server. It projects native APIs like creating dashboards on Grafana into Kubernetes resource APIs, so that users can manage 3rd-party resources as Kubernetes native resources.
• Vela Workflow: The workflow engine translates CUE-based steps and executes them. It works as a pure delivery tool and can be used aside by the KubeVela application. Compared to Tekton, it mainly organize the process in CUE style, instead of using Pods and Jobs directly.

Stability​

To ensure KubeVela is able to handle certain amount of applications under limited resources, multiple load testings have been conducted under various circumstances. The experiments have demonstrated that the performance of KubeVela system is capable of dealing thousands of applications in an ordinary-sized cluster. The observability infrastructure further exposes the bottleneck of KubeVela and guides system operators to do customized tunning to improve the performance in specific use environments.

In a nutshell

Currently, KubeVela has already been applied in production by a number of adopters from various areas. Some mainly use KubeVela's abstraction capability to simplify the use and deploy of applications. Some build application-centric management system upon KubeVela. Some use the customized workflow to orchestrate the delivery process. It is especially welcomed in high-tech industries and shown to be helpful for delivering and managing enourmous applications.

The KubeVela community has attracted world-wide contributors and continuously evolves over the past two years. Nowadays, there are over 200 contributors from various contries have participated in the developing of KubeVela. Thousands of issues have been raised and 85% of them are already solved. There are also bi-weekly community meetings held in both English and Chinese community.

With more and more people coming into the community, KubeVela is consistently upgrading itself to fit into more complex, varying use cases and scenarios.

Application Delivery on Kubernetes​

The cloud-native landscape is formed by a fast-growing ecosystem of tools with the aim of improving the development of modern applications in a cloud environment. Kubernetes has become the de facto standard to deploy enterprise workloads by improving development speed, and accommodating the needs of a dynamic environment.

Kubernetes offers a comprehensive set of entities that enables any potential application to be deployed into it, independent of its complexity. This however has a significant impact from the point of view of its adoption. Kubernetes is becoming as complex as it is powerful, and that translates into a steep learning curve for newcomers into the ecosystem. Thus, this has generated a new trend focused on providing developers with tools that improve their day-to-day activities without losing the underlying capabilities of the underlying system.

Napptive Application Platform​

The NAPPTIVE Playground offers a cloud native application platform focused on providing a simplified method to operate on Kubernetes clusters and deploy complex applications without needing to work with low-level Kubernetes entities. This is especially important to be able to accommodate non-developer users' personas as the computing resources of a company are being consolidated in multi-purpose, multi-tenant clusters. Data scientists, business analysts, modeling experts and many more can benefit from simple solutions that do not require any type of knowledge of cloud computing infrastructures, or orchestration systems such as Kubernetes, which enable them to run their applications with ease in the existing infrastructure.

Any tool that works in this space must start by analyzing the existing abstractions. In particular, at Napptive, we focus on the Applications. This quite understood abstraction is not present in Kubernetes, requiring users to manually tag, identify and reason about the different components involved in an application. The Open Application Model provides an excellent abstraction to represent any type of application independently of the cloud provider, containerization technology, or deployment framework. The model is highly customizable by means of adding Traits, Policies, or new Component Definitions.

KubeVela in Napptive​

The Napptive Playground leverages Kubevela as the OAM runtime for Kubernetes deployments. Our Playground provides an environment abstraction with multi-tenant guarantees that is equivalent to partitioning a shared cluster by means of differentiated namespaces. The benefit of our approach is the transparency of its configuration using a higher abstraction level that does not involve any Kubernetes knowledge.

The following diagram describes the overall architecture of Napptive and Kubevela deployed in a Kubernetes cluster.

The user has the ability to interact with the cluster by using the Napptive user interface (CLI or Web UI), or by means of using the standard Kubernetes API with kubectl. Isolated environments can be easily created to establish the logical separations such as type of environment (e.g., deployment, staging, production), or purpose (e.g., projectA, projectB), or any other approach to differentiate where to deploy an application. Once the OAM application is deployed on Kubernetes, Kubevela is in charge of managing the low-level application lifecycle creating the appropriate Kubernetes entities as a result. One of the main differences with other adopters of Kubevela, is the fact that we use Kubevela in a multi-tenant environment. The following figure shows the specifics of an application deployment in this type of scenario.

The Napptive Playground is integrated with Kubernetes RBAC and offers a native user management layer that works in both on-premise and cloud deployments. User identity is associated with each environment, and Kubevela is able to take that information to ensure that users can only access their allowed resources. Once a user deploys an OAM application, Kubevela will intercept the call and attach the user identity as an annotation. After that, the rendering process will ensure that the user has access to the entities (e.g., trait, component, policy, workflow step definitions) and that the target namespaces are also accessible by the user. Once the application render is ready, the workflow orchestrator will create the different entities in the Kubernetes namespace.

Napptive in the Community​

In terms of our involvement with the OAM/Kubevela community, it has evolved overtime from being passive members of the community simply exploring the possibilities of OAM in Kubernetes, to becoming active contributor members in different areas. In particular, we have closely worked with the core Kubevela development team to test and overcome the different challenges related to using a multi-tenant, RBAC compliance installation of Kubevela. Security in this type of installation is critical and it is one of our main focuses within the Kubevela community. We are committed to continue working with the community not only to ensure that multi-tenancy is maintained throughout the different releases, but also to add our own perspective representing our customer use cases.

The Kubevela community is growing at a fast pace, and we try to contribute our adaptations, features, feedback, and thoughts back to the community. This type of framework is deployed in a multitude of environments. The most common one is probably where one or more clusters are inside a company. In our particular application, we are interested in exploring methods to offer computing capabilities for a variety of user personas in a shared cluster, and we contribute our view and experience in the community.

Future Evolution​

In terms of future evolution, we believe the Napptive Playground is a great tool to experience working with the Open Application Model without the need of installing new clusters and frameworks. From the point of view of our contributions in the community, we are internally working in exploring QA mechanisms that ensure that customer applications remain working after upgrades, and identifying potential incompatibilities between releases. Once that work is ready, we plan to contribute our testing environment setup back to the community so that it can be adopted in the main branch. We are also excited with new functionalities that have been recently added to Kubevela such as multi-cluster support, and are exploring methods to adopt it inside the Playground. Moreover, we are actively working on an OAM-compatible application catalog that will simplify the way organizations store and make accessible application definitions so that they can be deployed into a cluster from a central repository. The catalog focuses on the OAM entities and relies on existing container registries for storing the image.

今天，阿里云云原生应用平台总经理丁宇在云栖大会隆重发布了 KubeVela 的全新升级！本次升级是 KubeVela 从应用交付到应用管理不断量变形成的一次质变，同时也开创了业界基于可扩展模型构建交付和管理一体化应用平台的先河。

KubeVela 1.5 于近日正式发布。在该版本中为社区带来了更多的开箱即用的应用交付能力，包括新增系统可观测；新增Cloud Shell 终端，将 Vela CLI 搬到了浏览器；增强的金丝雀发布；优化多环境应用交付工作流等。进一步提升和打磨了 KubeVela 作为应用交付平台的高扩展性体验。另外，社区也正式开始推动项目提级到 CNCF Incubation 阶段，同时在多次社区会议中听取了多个社区标杆用户的实践分享，这也证明了社区的良性发展。项目的成熟度，采纳度皆取得了阶段性成绩。这非常感谢社区 200 多位开发者的贡献。

你可能已经从这篇博客 中了解到，我们可以通过 terraform 插件使用 vela 来管理云资源（如 s3 bucket、AWS EIP 等）。 我们可以创建一个包含一些云资源组件的应用，这个应用会生成这些云资源，然后我们可以使用 vela 来管理它们。

有时我们已经有一些 Terraform 云资源，这些资源可能由 Terraform 二进制程序或其他程序创建和管理。 为了获得 使用 KubeVela 管理云资源的好处 或者只是在管理云资源的方式上保持一致性，我们可能希望将这些现有的 Terraform 云资源导入 KubeVela 并使用 vela 进行管理。如果我们只是创建一个描述这些云资源的应用，这些云资源将被重新创建并可能导致错误。 为了解决这个问题，我们制作了 一个简单的 backup_restore 工具。 本博客将向你展示如何使用 backup_restore 工具将现有的 Terraform 云资源导入 KubeVela。

如果您正在寻找将 Terraform 生态系统与 Kubernetes 世界粘合在一起的东西，那么恭喜！你在这个博客中得到了你想要的答案。

随着各大云厂商产品版图的扩大，基础计算设施，中间件服务，大数据/AI 服务，应用运维管理服务等都可以直接被企业和开发者拿来即用。我们注意到也有不少企业基于不同云厂商的服务作为基础来建设自己的企业基础设施中台。为了更高效，统一的管理云服务，IaC 思想近年来盛行，其中 Terrafrom 更是成功得到了几乎所有的云厂商的采纳和支持。以 Terrafrom 模型为核心的云服务 IaC 生态已经形成。然而在 Kubernetes 大行其道的今天，IaC 被冠以更广大的想象空间，Terraform IaC 能力和生态成果如果融入 Kubernetes 世界，我们认为这是一种强强联合。

KubeVela 是一个现代化的软件交付控制平面，目标是让应用的部署和运维在如今的混合多云环境下更简单、敏捷、可靠。自 1.1 版本发布以来，KubeVela 架构上天然打通了企业面向混合多云环境的交付难题，且围绕 OAM 模型提供了充分的可扩展性，赢得了大量企业开发者的喜爱，这也使得 KubeVela 的迭代速度不断加快。

1.2 版本我们发布了开箱即用的可视化控制台，终端用户可以通过界面发布和管理多样化的工作负载；1.3 版本 的发布则完善了以 OAM 模型为核心的扩展体系，提供了丰富的插件功能，并给用户提供了包括 LDAP 权限认证在内的大量企业级功能，同时为企业集成提供了巨大的便利。至今为止，你已经可以在 KubeVela 社区的插件中心里获得 30 多种插件，其中不仅包含了 argocd、istio、traefik 这样的 CNCF 知名项目，更有 flink、mysql 等数据库中间件，以及上百种不同云厂商资源可供直接使用。

在这次发布的 1.4 版本中，我们围绕让应用交付更安全、上手更简单、过程更透明三个核心，加入了包括多集群权限认证和授权、复杂资源拓扑展示、一键安装控制平面等核心功能，全面加固了多租户场景下的交付安全性，提升了应用开发和交付的一致性体验，也让应用交付过程更加透明化。

随着云原生的不断发展和成熟，越来越多的基础设施能力逐渐标准化成为 PaaS 平台或者 SaaS 化产品。一个产品的诞生不再像过去那样需要建立一个团队，从开发、测试一直到运维、基础设施全部分多种角色系统完成。如今，敏捷组织文化和云原生技术驱动，使得这些职责更多的是“左移”到了开发者身上，测试左移、监控左移、安全左移，以及 DevOps 等一系列理念都是在强调，通过开源项目或者云的产品和服务将测试、监控、安全、运维等一系列事务提前到开发阶段完成。这看似美好的愿景却给开发者带来了巨大的挑战，开发者对底层五花八门的产品和复杂 API 缺乏掌控力，他们不仅仅是在做选择，更多的需要去理解和协调底层复杂异构的基础设施能力，以便满足上层业务的快速发展和迭代需求。

这种复杂性和不确定性无疑大大降低了开发者的体验，降低了业务系统的交付效率，增加了运维风险。开发者体验的核心是“简单”和“高效率”，不管是开发者还是企业都需要更好用的开发者工具或者平台来达成。在现代云原生技术之上打造一款帮助开发者从开发、交付以及后续持续运维的一体化平台，一直是 KubeVela 演进的核心目标。如图 1 所示，在 v1.2 版本中，我们围绕开发者体验新增了 UI 控制台组件（VelaUX），简化了编排 YAML 的复杂性，完善了插件体系建设，丰富了云资源的扩展能力，增加了大量 CI/CD 等生态对接的能力，进一步完善了开发者端到端的使用体验。

图1：KubeVela 架构设计

在云原生理念迅速普及的今天，混合环境部署（混合云/多云/分布式云/边缘）已经成为了大多数企业应用、SaaS 服务、应用持续交付平台的必然选择，而云原生技术的发展趋势也正在朝着“一致的、跨云、跨环境的的应用交付”不断迈进。然而，无论是 Kubernetes 本身还是现有的各类应用交付系统，都没有在现今混合、分布式的部署环境之上引入一致的上层抽象来为应用交付进行建模。这种缺乏统一上层抽象的应用交付过程，往往同底层基础设施紧密耦合，导致用户心智负担很重并且严重依赖于用户个人的经验和能力。这不仅会大幅影响用户体验、降低生产效率，甚至还会导致错误和故障的发生。

而现在，这个问题终于有了一个开源、标准，又不失灵活度的解法。它就是：

KubeVela 作为一个开箱即用、面向现代微服务架构的应用交付与管理平台，今天正式发布了 1.1 版本，以更加用户友好和完善的功能集，开启了”让混合环境应用交付更加简单高效“的重要里程碑。