sandbox

sandbox copied to clipboard

Application contact emails

• [email protected]
• [email protected]
• [email protected]

Project Summary

A technology stack for building cloud-native Internal Developer Platforms (IDPs)

Project Description

What it does

KusionStack is a technology stack for building cloud-native IDPs. It enables application developers to perform all operational tasks throughout the DevOps lifecycle in one place, using one environment-agnostic configuration with building blocks, across multiple different infrastructures such as Kubernetes, clouds and on-premises infrastructures.

The building blocks are defined by platform engineers, designed to hide the infrastructure complexity while only exposing simple and developer-friendly schemas to the application developers, in order to reduce their cognitive overhead from the infrastructure concepts. The platform-standardized configurations such as security and compliance best practices are also codified into or serving as inputs to these building blocks.

Based on this design, KusionStack defines a new paradigm for application developers and platform engineers to collaborate. With the separation of concerns, different roles are focused on their parts based on their expertise and responsibility.

In addition, we are continuously adding components to KusionStack to provide a more secure and efficient path to build an IDP. For instance, operating and controller-mesh under KusionStack intend to enhance Kubernetes operational security, which help users build a more secure Kubernetes-based IDP.

Why it's needed

Cloud-native technologies are evolving constantly, delivering immense values but in the meantime, introducing new challenges to software organizations. The variety of infrastructures has exploded, significantly increasing the complexity of application delivery and operations. As the infrastructure continues to expand, developers face a rapidly multiplying cognitive overhead. In the meantime, the platform teams can't keep up with the pace of infrastructure development, making the platform a potential efficiency bottleneck. The traditional "ticketOps" approach is no longer suitable and we need a new way to navigate through the DevOps lifecycle of applications.

Org repo URL (provide if all repos under the org are in scope of the application)

https://github.com/KusionStack

Project repo URL in scope of application

https://github.com/KusionStack/kusion

Additional repos in scope of the application

https://github.com/KusionStack/karpor https://github.com/KusionStack/operating https://github.com/KusionStack/controller-mesh

Website URL

https://kusionstack.io/

Roadmap

Kusion: https://www.kusionstack.io/docs/reference/roadmap Karpor: https://www.kusionstack.io/karpor/roadmap/

Roadmap context

This RoadMap listed above is at a high level and it's by each product under KusionStack.

Contributing Guide

https://github.com/KusionStack/community/blob/main/CONTRIBUTING.md

Code of Conduct (CoC)

https://github.com/KusionStack/community/blob/main/CODE_OF_CONDUCT.md

Adopters

https://github.com/KusionStack/community/blob/main/USERS.md

Contributing or Sponsoring Org

Ant Group

Maintainers file

https://github.com/KusionStack/community/blob/main/MAINTAINERS.md

IP Policy

• [X] If the project is accepted, I agree the project will follow the CNCF IP Policy

Trademark and accounts

• [X] If the project is accepted, I agree to donate all project trademarks and accounts to the CNCF

Why CNCF?

Alignment: As a Platform Engineering advocate, KusionStack has the vision to address challenges within applications' full DevOps lifecycle (delivery, operations, etc) in a time of aggressively expanding infrastructure technologies, and consequently, the cognitive burden for the developers that comes with it. KusionStack aims to eliminate infrastructure complexity for cloud-native applications and enable developer self-service via disciplines of Platform Engineering, which aligns perfectly with CNCF's mission to make cloud-native technologies ubiquitous.

Community: The CNCF hosts a large and vibrant community of developers and users, which can adopt, drive innovation, and contribute to KusionStack. Becoming a part of CNCF would help KusionStack attract more attention and developers, enrich the ecosystem, and accelerate growth.

Guidance and Governance: We wish to receive guidance from CNCF in building a healthy and dynamic community that helps KusionStack grow. In addition, adhering to the CNCF standards and interoperability makes it easier to integrate with other well-liked cloud-native technologies, which drives adoption.

Benefit to the Landscape

Help to build the Platform Engineering community: Being an early Platform Engineering practitioner, KusionStack is incubated at AntGroup based on years of cloud-native practice in production at a massive scale. The CNCF is also promoting and building a platform engineering community and has authored a platform whitepaper. We hope to join this community, contribute our efforts, and provide viable Platform Engineering solutions for the community with an aligned mission.

Simplify and drive adoption for other CNCF Projects: KusionStack has already integrated with several projects in the CNCF ecosystem. KusionStack aims to reduce if not eliminate the complexity of using cloud-native technologies via disciplines of Platform Engineering, which can drive adoption for other CNCF projects. Having such a toolset provides relevant users a leverage to harvest the full power of cloud-native technologies. With the ongoing effort to further integrate into the CNCF world, KusionStack will serve a broader range of scenarios, helping more users build more advanced platforms based on the principles of the CNCF platform whitepaper.

Cloud Native 'Fit'

Automation & Configuration

Cloud Native 'Integration'

As a tech stack to build an IDP, KusionStack is designed to minimize the friction in the DevOps lifecycle of cloud-native applications, ranging from application delivery to day-2 operations, which includes integration with a wide spectrum of cloud-native technologies.

In theory, KusionStack is technology neutral - In that it encapsulates a collection of tooling that enables the assembling of a golden path that meet the specific needs of its users (it can be either an individual user or an organization). The capabilities are modularized by design and can be extended with minimum effort.

In practice, KusionStack will prioritize integration for the most common needs in the lifecycle of cloud-native applications, such as:

• Workload management: Kubernetes is considered the current de-facto standard for the compute components and the primary compute platform KusionStack is currently prioritized on. Any other toolings (such as KubeVela, Crossplane, Ingress-controllers, cert-manager, etc) within the Kubernetes ecosystem that follow Kubernetes Resource Model (KRM) specification can be leveraged via the generator mechanism in Kusion which generates the manifest for said resources.
• Infrastructure provisioning: KusionStack leverages Terraform providers to provision infrastructure on clouds, both public and private. On top of that, one of the core values KusionStack provides is a single definition(manifest) and a uniform workflow for both application workloads and infrastructure - including automatically connecting them to create a seamless experience for the user.
• Monitoring and observability - Prometheus
• Traffic Management: Istio (planned)
• Secret Management: Vault
• Configuration Management: KCL
• Policy Management: OPA (planned)
• Security Scanning: Kube-bench, Kube-hunter (planned)
• etc

The extendability of KusionStack (and therefore the integration mechanism with existing cloud-native technology) is mostly reflected in Kusion Modules. Kusion Modules are building blocks of re-usable code that represents a set of abstracted capabilities. Kusion Modules are defined by platform engineers and leveraged by the end user. We will also ship some common capabilities out-of-the-box.

Cloud Native Overlap

From the perspective of technical philosophy, product pattern and ultimate goal, we haven't found a significant overlap between KusionStack and other existing CNCF Projects. Although some might initially perceive an overlap between Kusion and KubeVela, they are in fact complementary and can be integrated to work together. As a lightweight, purely client-side tool, coupled with the corresponding Generator implementation, Kusion can render application configuration models to generate CRD resources for KubeVela and leverage KubeVela's control plane to implement application delivery.

KusionStack provides some of the foundational tools necessary for building an internal developer platform, but building a complete internal developer platform requires additional ecosystem support, such as infrastructure as code tools, CI/CD pipelines, GitOps engine, which fall outside the scope of KusionStack, also the community provides very mature technical solutions in these areas. Platform engineers can combine all these technologies to construct a truly production-ready IDP platform.

Similar projects

KubeVela

Landscape

Yes

Business Product or Service to Project separation

N/A

Project presentations

N/A

Project champions

N/A

Additional information

N/A

Roadmap

https://www.kusionstack.io/docs/kusion/reference/roadmap

link changed: https://www.kusionstack.io/docs/reference/roadmap

TAG-CS review, this project has:

• A short WIP contributing guide
• The beginnings of a contributor ladder, but no other documented goverance
• 10 maintainers, not documented as to affiliation

TAG-CS review, this project has:

• A short WIP contributing guide
• The beginnings of a contributor ladder, but no other documented goverance
• 10 maintainers, not documented as to affiliation

@jberkus Thanks for the feedback! We'll take these as the action items. Also I'm noticing KusionStack is listed with the status "Upcoming" in the Sandbox Application Board - next review June 11. Is there anything (additional materials/information) we need to prepare prior to the date?

Nope! If there are questions, the TOC will send them to you or post them in this issue.

@SparkYuan thank you for applying for CNCF Sandbox! The TOC would like to understand more about the project - please coordinate a presentation with TAG App Delivery and a recommendation from the TAG before the next Sandbox review in August.

@lianmakesthings @joshgav @thschue Has TAG App Delivery discussed and reviewed the project to provide a recommendation for the TOC?

@lianmakesthings @joshgav @thschue Has TAG App Delivery discussed and reviewed the project to provide a recommendation for the TOC?

We have coordinated with TAG for a presentation session at the next meeting on July 10th. cc @angellk

Slides from yesterday's presentation: https://drive.google.com/file/d/1XZcVpfF95G1o1NChACjEgPWyiUb41NQd/view?usp=sharing

Hi @angellk, just want to follow up on this. We have presented to the App-Delivery TAG last week on July.10th. You can find the slides here. I noticed this issue has been moved to Upcoming. In the meantime, is there anything you need from us to move this forward? Thanks!

Looks like the project's roadmap has been moved, I was able to find the correct roadmap linked here: https://www.kusionstack.io/docs/reference/roadmap

The adopters file linked is also invalid, this appears to be the correct file: https://github.com/KusionStack/community/blob/main/USERS.md

Vault is a non-CNCF project. Does KusionStack have plans to integrate with Open-Bao? I see support for other commercial secrets mgmt solutions.

For Compliance Report of Karpor, what are the sources of the compliance findings? Are they configurable? How often are they updated?

looking at the other projects under KusionStack, they appear to be in varying states of maturity (using completeness of documentation as an indicator). Does KusionStack have plans to define levels for each of these?

Hi @TheFoxAtWork, thanks for pointing out the outdated information! We have updated the links accordingly.

Vault is a non-CNCF project. Does KusionStack have plans to integrate with Open-Bao? I see support for other commercial secrets mgmt solutions.

We can absolutely take a look at that. KusionStack itself is technology neutral. It’s designed to let platform engineers come in and define their preferred way to connect with ANY infrastructure, to eventually build a tailored golden path that meets specific organizational needs.

For Compliance Report of Karpor, what are the sources of the compliance findings? Are they configurable? How often are they updated?

The reports are generated from third-party, open-source tools. Currently it has built-in support for a tool called kubeaudit which supports custom auditors. We have more third-party tools planned in the future, which extends the risk exposure beyond static YAML scanning to include things like CVE scanning, CIS Benchmarking, etc. Yes they will be configurable in the future. The reports are designed to be cached for 10 minutes by default and can be re-generated anytime on the spot.

looking at the other projects under KusionStack, they appear to be in varying states of maturity (using completeness of documentation as an indicator). Does KusionStack have plans to define levels for each of these?

Excellent question. We actually would like to discuss this a bit as well.

KusionStack originates from years of experiences building platforms at scale at Ant Group. We are currently using KusionStack as an overarching group of platform engineering products we want to open source (i.e. stuff we believe that can be helpful for platform owners to build an IDP), including a Platform Orchestrator (Kusion), an insights & intelligence product (Karpor), a set of Kubernetes controllers solutions (operating & controller-mesh), and in the future, an IAM-related and perhaps a change-related product. They are designed to be loosely coupled but synergizes well when used together.

So as you noticed, there’s a difference in maturity levels for these products because they are in different stages in their product lifecycle. Kusion is seen as the centerpiece and what we consider to be the most production-ready at this time. As a result, a significant chunk of our presentation is about Kusion as the platform orchestrator. And then we have other products that are proven to be valuable internally but obviously needs more polishing going into the community. We are more than happy to define their maturity levels individually and draw these lines somewhere if these differences are deemed problematic for its community success.

As an extension to that statement, our question is, does the community generally welcome a larger initiative like this, or does it prefer smaller ones, i.e. break it apart into smaller pieces to have more certainty in each? And also from your experience, in terms of PLG, does either option presents a route that's more likely setting up the project for success?

@ffforest thanks for the quick responses!

I'd like to get better clarification on the separation of the described "Products" from the Open Source projects detailed here. I think that these technologies are what is currently used by Ant Group for their internal platforms and not currently products for purchase/subscription from Ant Group by non-Ant Group entities - Is this correct?

Regarding:

does the community generally welcome a larger initiative like this, or does it prefer smaller ones, i.e. break it apart into smaller pieces to have more certainty in each? And also from your experience, in terms of PLG, does either option presents a route that's more likely setting up the project for success?

The TOC sees several different patterns for projects that apply or are currently under the CNCF.

• Singular projects that are clearly defined and may address a limited set of use cases in cloud native

• Large projects that address larger use cases or integrate technologies to solve multiple use cases. These projects often include sub-projects that are part of the overall project and critical to the functionality and use case implementation the Project seeks to address. Sub-projects are often packaged as part of the core project due to their criticality in the operation of the core project.

• Large projects that address large use cases with supporting sub-projects for additional benefit. These projects typically have a "core" project that serves the primary defined use cases. They also contain other sub-projects that support, extend, and enhance the core project or may be used independent of the core project. Those sub-projects are often packaged independent of the core project.

For the TOC's processes, if the project is substantially large - sub-projects not packaged with the "core" project are evaluated against the defined Project governance for managing them and their maturity and therefore the responsibility of the project to hold them accountable to that stated governance.

For contributors, I cannot attest to their experience. @jberkus @cathpag @geekygirldawn : Do you have observations on this topic you can share? higher probability of contributions and engaged community in smaller or larger projects? I imagine it largely depends on the project governance.

We have a few projects in the CNCF that are "federated" projects with multiple separable subprojects. Our experience is that the majority of contributors end up whichever of those subprojects becomes the most popular; Argo and Konveyor are both examples of this. So it's important for the core project to have plans for what to do if some of the subprojects "fall behind".

Hi @TheFoxAtWork and @jberkus, appreciate the answer!

I think that these technologies are what is currently used by Ant Group for their internal platforms and not currently products for purchase/subscription from Ant Group by non-Ant Group entities - Is this correct?

Yes that is correct. Here is a non-comprehensive list of companies that are using the open source version of kusion but there are no commercial activities if that's what you are asking.

For the TOC's processes, if the project is substantially large - sub-projects not packaged with the "core" project are evaluated against the defined Project governance for managing them and their maturity and therefore the responsibility of the project to hold them accountable to that stated governance.

Gotcha. You can find KusionStack's project level governance in the community repo. From an action standpoint, is there anything else we could provide in the meantime?

Forest - thanks for the clarification!

for now, as TAG, Community, and TOC have questions about the project, just answering them as you have time before the review in August. thank you for being so responsive!

Hi @ffforest thanks for submitting your application! I have the following questions while reviewing your project:

1. Has there been any effort to standardize the AppConfiguration Spec?
2. How is the contribution look like outside of Ant group? Currently all maintainers are from Ant group: https://github.com/KusionStack/community/blob/main/MAINTAINERS.md
3. Do you have a sample workspace yaml or its API spec? I could not find one. I'm curious learning a bit more.

Thanks!

Hi @ffforest thanks for submitting your application! I have the following questions while reviewing your project:

1. Has there been any effort to standardize the AppConfiguration Spec?
2. How is the contribution look like outside of Ant group? Currently all maintainers are from Ant group: https://github.com/KusionStack/community/blob/main/MAINTAINERS.md
3. Do you have a sample workspace yaml or its API spec? I could not find one. I'm curious learning a bit more.

Thanks!

Hi @linsun,

1. Depends on what you mean by "standardize". AppConfiguration is our take on how to describe developer's intent when shipping an application. Its definition can be found here written in KCL. We have been iterating it and intend to use it as the standardized developer interface within KusionStack, including those use cases inside Ant Group. But if you are talking about making it another OAM or Score, that's currently not the plan.
2. KusionStack has 13 non-Ant-Group contributors over the last 6 months. Kusion has 5 (vietanhtwdk, hoangndst, skoenig, EdenBW, ekjotsinghmakhija) and Karpor has 8 (virtually-stray, Paradiesvogel7, solarhell, jueli12, rajeshkio, JasonHe-WQ, z1cheng, CirillaQL). They can choose to become the MAINTAINER once they are more involved in the project. The defined community roles can be found here.
3. Absolutely. Here is a sample workspace.yaml for an AI Agent demo we did at PlatformCon 24. And here's the corresponding main.k(developer side config). If you are also interested in the demo, it can be found here.

Hi @ffforest thanks for submitting your application! I have the following questions while reviewing your project:

1. Has there been any effort to standardize the AppConfiguration Spec?
2. How is the contribution look like outside of Ant group? Currently all maintainers are from Ant group: https://github.com/KusionStack/community/blob/main/MAINTAINERS.md
3. Do you have a sample workspace yaml or its API spec? I could not find one. I'm curious learning a bit more.

Thanks!

Hi @linsun, @ffforest has answered your questions, and I'd like to add some additional points regarding the third question. We've recently updated the explanation of workspace on our website. You can find more details, including an example here

The scope of this set of projects includes a number of areas including idp mesh and others. This raises a concern about accepting this project as defined as it's difficult to determine if the scope of the submission will change further over time.

Can you provide some clarity around the roadmap and intended scope of kusionstack going forward?

The scope of this set of projects includes a number of areas including idp mesh and others. This raises a concern about accepting this project as defined as it's difficult to determine if the scope of the submission will change further over time.

Can you provide some clarity around the roadmap and intended scope of kusionstack going forward?

Hi @mauilion, I would characterize the intended scope of KusionStack as "tools that would be helpful for platform engineers to build an IDP that suits their needs". It's not a platform but rather a set of loosely coupled tools to help you get there. We are not expecting this vision to change long term because we believe internal platforms should be situationally tailored to their organizational needs.

KusionStack first started with kusion and kcl (another sandbox project), which evolved to be platform orchestrator and config management DSL respectively. But that's not the only thing important when building an IDP. Throughout our platform engineering journey, we built out additional components in our IDP, including customized Kubernetes workloads and operators, and a multi-cluster data plane with intelligence (karpor), and decided to open source them because they were great additions to the platform. In the future we intend to open source an IAM-related and a change-related product as well.

The roadmaps provided in this application refer to the individual ones for each of the tools mentioned above. Additionally there is a KusionStack-wide intro you can find here. It's basically what I said above but I'd be more than happy to consolidate them into a KusionStack-wide roadmap.

The scope of this set of projects includes a number of areas including idp mesh and others. This raises a concern about accepting this project as defined as it's difficult to determine if the scope of the submission will change further over time.

Can you provide some clarity around the roadmap and intended scope of kusionstack going forward?

Hi @mauilion, I’d like to add a little more on this subject. If this helps visualizing, this is our current interpretation of what building an IDP entails (diagram inspired by Pulumi), which sort of outlines the broad scope of what KusionStack intends to include in the future.

Our vision is to enable platform builders to define opinionated golden paths rather than shipping a platform ourselves. That will be sort of the anchor during the development of this project.

I have also added a KusionStack-level roadmap that includes the features to drive each project forward, grouped by their maturity levels.

Also this might be a longshot but I’d be more than happy to talk over this at KubeCon HongKong next week if anyone is coming.

Hi there, I'm checking to see if there any update on this issue. Or should we expect an update on the next review date Oct.8th? cc @mauilion @TheFoxAtWork

@ffforest apologies for the delay, I hope to have an update for you by end of this week.

Moving to a vote :+1: /vote

Vote created

@mrbobbytables has called for a vote on [Sandbox] KusionStack (#83).

The members of the following teams have binding votes:

Team
@cncf/cncf-toc

Non-binding votes are also appreciated as a sign of support!

How to vote

You can cast your vote by reacting to this comment. The following reactions are supported:

In favor	Against	Abstain
👍	👎	👀

Please note that voting for multiple options is not allowed and those votes won't be counted.

The vote will be open for 2months 30days 2h 52m 48s. It will pass if at least 66% of the users with binding votes vote In favor 👍. Once it's closed, results will be published here as a new comment.

/check-vote

Vote status

So far 54.55% of the users with binding vote are in favor (passing threshold: 66%).

Summary

In favor	Against	Abstain	Not voted
6	0	0	5

Binding votes (6)

User	Vote	Timestamp
kevin-wangzefeng	In favor	2024-09-03 16:04:06.0 +00:00:00
TheFoxAtWork	In favor	2024-09-03 16:46:46.0 +00:00:00
mauilion	In favor	2024-09-03 15:07:07.0 +00:00:00
cathyhongzhang	In favor	2024-09-03 15:48:10.0 +00:00:00
linsun	In favor	2024-09-03 15:48:47.0 +00:00:00
angellk	In favor	2024-09-03 17:00:58.0 +00:00:00
@dims	Pending
@rochaporto	Pending
@dzolotusky	Pending
@nikhita	Pending
@kgamanji	Pending

Non-binding votes (5)

User	Vote	Timestamp
pacoxu	In favor	2024-09-04 1:08:20.0 +00:00:00
adohe	In favor	2024-09-04 2:30:41.0 +00:00:00
ffforest	In favor	2024-09-04 3:47:32.0 +00:00:00
SparkYuan	In favor	2024-09-04 6:11:08.0 +00:00:00
hoangndst	In favor	2024-09-04 23:43:33.0 +00:00:00

/check-vote

Vote status

So far 54.55% of the users with binding vote are in favor (passing threshold: 66%).

Summary

In favor	Against	Abstain	Not voted
6	0	0	5

Binding votes (6)

User	Vote	Timestamp
mauilion	In favor	2024-09-03 15:07:07.0 +00:00:00
kevin-wangzefeng	In favor	2024-09-03 16:04:06.0 +00:00:00
cathyhongzhang	In favor	2024-09-03 15:48:10.0 +00:00:00
linsun	In favor	2024-09-03 15:48:47.0 +00:00:00
TheFoxAtWork	In favor	2024-09-03 16:46:46.0 +00:00:00
angellk	In favor	2024-09-03 17:00:58.0 +00:00:00
@dims	Pending
@rochaporto	Pending
@dzolotusky	Pending
@nikhita	Pending
@kgamanji	Pending

Non-binding votes (5)

User	Vote	Timestamp
pacoxu	In favor	2024-09-04 1:08:20.0 +00:00:00
adohe	In favor	2024-09-04 2:30:41.0 +00:00:00
ffforest	In favor	2024-09-04 3:47:32.0 +00:00:00
SparkYuan	In favor	2024-09-04 6:11:08.0 +00:00:00
hoangndst	In favor	2024-09-04 23:43:33.0 +00:00:00

/check-vote