cncf
[Initiative]: Specification for declaring application integration dependencies
Name
Specification for declaring application integration dependencies
Short description
Spec that lists integration dependencies needed for an application to run successfully, e.g. DBs, message bus, etc.
Responsible group
TAG Developer Experience
Does the initiative belong to a subproject?
No
Subproject name
No response
Primary contact
Colin Lacy ([email protected])
Additional contacts
Graziano Casto ([email protected]) Alexandra Aldershaab ([email protected])
Initiative description
One of the common things we hear in software development is that if we package an application a certain way, it can run anywhere. While that may be true in terms of executing the command that runs it, it doesn’t guarantee that the application won’t exit 1 within a few milliseconds when it can’t connect to a database or cache, or start throwing 500s when it tries to call to a peer service that hasn’t been deployed. Yes, it can run, but can it run successfully?
This initiative would set out to create a project-agnostic specification that for declaring a service’s dependencies, so that it could indicate to an application deployment team what other services must be readily available for the application to run successful - e.g. APIs, databases, caches, blob stores, message buses, etc. This is not meant to replace conversations between application engineers and production support. Rather, it’s meant to give them a standardized reference that they can use to enable their discussion.
I say project agnostic specification because there have been attempts to build this as part of a solution in the past, e.g. Radius. Since the Radius approach packaged the spec in with the solution, the spec has only evolved to the degree that the solution can support it, most notably with a strong focus on Azure services. Making the spec project-agnostic would create a solution-independent language that can be implemented however necessary by conforming projects, including Radius. In fact, this effort would likely look to Radius, Porter, and other projects like them for inspiration. But the benefit of a spec is that it could also be leveraged for other use cases, such as guaranteeing abstraction coverage in Dapr, or for integration mocking in Microcks. Furthermore, it can support security efforts that are focused on expected application behavior, such as Software Bill of Behavior (SBOB) initiatives.
Deliverable(s) or exit criteria
- first deliverable is research and report on existing specs and previous attempts that have failed (and maybe could be resurrected) to better understand the problem space
- the outcome of this first step will dictate the direction of subsequent steps, but the following should still apply regardless
- this is the first potential exit point if, for whatever reason, the conclusion in the report is that this is too complex and we recommend that a spec not be created
- after the report has been delivered to the TAG, identify (and potentially extend) and existing spec, or as a last resort create a specification that indicates what services must be available to an application to ensure that it can run successfully
- First release will be scoped to a specific integration set, e.g. external APIs, or required data stores (DB, cache)
- TBD based on group discussion
- The specification must exist in a format that can be consumed programmatically by spec consumers, e.g. JSON, YAML, etc.
- First release will be scoped to a specific integration set, e.g. external APIs, or required data stores (DB, cache)
- A plan and project description for automatic generation of dependency declarations that adhere to the spec during application development
- to be clear, this is the description or recommendation of a project, not the project itself
- Guidance for consumers of dependency declarations for how to consume the spec and interpret it for downstream use
- likely in the form of a short white paper
Expected timeline is 3-6 months.
My two cents @colinjlacy ..
I really like the topic of this initiative, but I see a few of things that might help us to be more focused:
- Creating a specification, is hard and usually I would recommend to figure out which projects can directly benefit from something like this. You mentioned Radius and Porter, but would this be useful for CD projects? Can they use this to validate before deploy? What about policy engines like Kyverno or OPA?
- What about creating an example for this?, with just plain Kubernetes to demonstrate the problem to be solved.
- Replace Dapper for
Dapr
@salaboy thanks for the feedback. My thoughts:
- It's pretty open, so I'd say that yes, policy engines like OPA and Kyverno, as well as Cilium with its
CiliumNetworkPolicyspec that could - in theory - be autogenerated from a clear spec to match expected behavior. CD projects could likely leverage this spec to ensure that dependencies exist before deploying. Both are solid points, thanks for bringing them up. - I'm not sure about the implementation yet, which is why I'd start with the spec. Maybe it could influence existing implementations - that is, for example, influence how Radius consumes a standardized dependency set. It's something that I think is certainly doable, but I'd look for help in putting together an implementation.
- Sorry! It's corrected above. Thanks for pointing that out.
Ok, so my suggestions (two cents) as we discussed in the meeting:
- Create a list of possible projects interested in this idea. This will help us to find the right people to have a scoped discussion
- Create some use cases description to be able to drive conversations and guide discussions.
- Having these list of projects and people can help us to define and scope the initiative with concrete outputs for the projects.
When I think about specs, I think about CDEvents Spec: https://cdevents.dev, which was created as an extension for CloudEvents but specifically for CI/CD use cases. For that case, the target projects were obvious, but Application Integration Dependencies can affect a wide range of projects and identifying which ones of those will be more interested in the idea will help us to prioritise work.
Hi @colinjlacy and team, I would love to hear your thoughts and feedback about where the existing Score (score.dev) as CNCF Sandbox does fit within this initiative. Score is first, a Workload spec, and then have 2 implementations (score-compose and score-k8s).
Looking forward to seeing what this initiative will generate, and the learnings and feedback that the Score project could benefit from it.
The positioning of other existing projects like Radius and KubeVela would be also interesting to see.
@mathieu-benoit thanks for taking a look! Ultimately the goal would be a tool-agnostic spec, but I'd love to see it build on the learning points from projects that have already solved problems in this space. My thought is that if it's tool agnostic, it can benefit app deployment, integration mocking, and security in a single declarative model. Tools like Score, Microcks, and Cilium could then consume that declaration file to power their own interpretations of what should happen downstream for specific use cases.
Slack channel for this issue: https://cloud-native.slack.com/archives/C09S7A5T3GF Running Google doc for notes: https://docs.google.com/document/d/1oz_1K1l-VuLuy-JPOh7oWrDqPzoImpYH5racSMJ5Br0/edit
Looks good to me! This should be ready to vote!
@salaboy @SwEngin @julsemaan @kdubois @cloudmelon @joshuabezaleel Feel free to add your thoughts too!
Made additional clarifications on the deliverables per recommendation made by Yacine in the Slack channel.
Made additional clarifications on the deliverables per recommendation made by Yacine in the Slack channel.
Thanks, @colinjlacy, and I agree regarding your comments and clarifications based on the pre-work we all need to do regarding this topic.
Looks like a great initiative! As mentioned by a few others, I'm not sure about wether we'll be able to get enough projects on board to get such a spec nailed down and broadly adopted but I think it's definitely worth a shot and I'm 100% in favor of this initiative.
I share the same thoughts as @kdubois here but 100% in favor of moving this forward and see if there is traction and desire for such as spec.
Agreed with others as well for moving this forward. And Colin has put a great effort in gathering the right people for the start.
As someone that is involved with Porter (which evolved from CNAB) for a little while, I was ecstatic when seeing other specification initiative sprung up like Radius and others, but I could see myself get confused if I were a software developer wanting to get started on which to choose, as I think (I could very well be wrong) not much conversation has happened between the said projects and teams in the past. Hence this could be a great momentum.
This proposal strongly reminds me of Cloud Foundry’s service binding model and the Service Broker API. It solved a similar dependency declaration problem inside a single, opinionated platform. Kubernetes, however, is a much more heterogeneous ecosystem, which means the problem space is dramatically larger.
Because of that, a cross-project, cross-vendor ‘dependency declaration spec’ is too broad. It spans infra provisioning, runtime guarantees, connectivity semantics, API contracts, eventing, and more. I don’t think a single spec can meaningfully cover this in 3–6 months, and it risks going in all directions at once.
I recommend narrowing the scope to one specific slice, for example, runtime startup requirements, or machine-readable API dependency declarations. That keeps the initiative realistic while still high-impact.
@SwEngin you're absolutely right about the scope risking being too big. I should have edited it to include this already, but in the kickoff meeting at KubeCon we discussed scoping it exactly the way you described - e.g. API integrations first, then DB integrations, then seeing how we feel after that. I'll update the goals list to reflect this.
You're also 100% correct about the Service Broker API. That was brought up in the issue's Slack channel (with a talk about why it failed). We want to learn from those projects to prevent history from repeating itself. Good call out!
Similarly, the goal here is not to create a spec if we don't have to. We are looking at the OCM spec and @joshuabezaleel brought up the CNAB spec as well. If we can leverage and/or extend those without having to start from scratch, all the better.
@riaankleinhans Thanks for moving forward with the vote status!
- Shouldn't you be assigned to this? It's still no assignee.
- How do we start with voting?
@danieloh30 as the voting and approval of the initiative are delegated to the TAGs, it can be an informal vote amongst the TAG leads and teach leads. Comment on the issue once the leads approved the initiative and I can move it along to the approved column.
@danieloh30 as the voting and approval of the initiative are delegated to the TAGs, it can be an informal vote amongst the TAG leads and teach leads. Comment on the issue once the leads approved the initiative and I can move it along to the approved column.
Gotcha
@salaboy @SwEngin @kdubois @julsemaan @joshuabezaleel @cloudmelon @graz-dev Time to vote! Give us a 👍 for your selection.
@SwEngin @danieloh30 @kdubois I believe it's up to one of you to make the official call to move this to Accepted. Could one of you please raise that request?
Sure! I accepted! @riaankleinhans can you please move forward to the accepted status with all leaders' comments?