operator-sdk

operator-sdk copied to clipboard

Feature Request

Is your feature request related to a problem? Please describe. There are several frameworks out there that aim at making it easier to develop operators by providing implementations for common tasks such as:

• https://github.com/redhat-cop/operator-utils
• https://github.com/RHsyseng/operator-utils
• https://github.com/halkyonio/operator-framework
• ...

Describe the solution you'd like It would be interesting to investigate these frameworks and see which features (if any) could be included in operator-sdk to make it easier to develop operators.

Proposed contributions

• [ ] Composable operator API: #2340

Should this be part of operator-framework/enhancements instead?

@metacosm thanks for bringing up these projects and the meta-issue of adding operator utilities to the SDK. If there are specific bits from a project that someone finds useful they should make a PR against the relevant operator-framework repo, ex. the SDK.

I would say an enhancement proposal is warranted if the contribution is a substantial chunk of/an entire repo. Did you have any specific features in mind?

I’ve started doing a comparative analysis of the different projects to see where they converge and what different things they cover. Depending on what might make sense for inclusion, it might end up being significant or not…

I think we can collect the feature additions relevant to SDK here for now. Maybe just keep a running checklist in the description of features we want to consider for inclusion?

Two other important points that are relevant here:

1. We're working on integrating with Kubebuilder, which will result in operator-sdk using Kubebuilder project layouts for Go operators. If any of the features are related to project files or project layout AND they would be applicable to ALL Go operators, we should consider adding the features to Kubebuilder instead of operator-sdk
2. We also heavily use controller-runtime, which is the library on which kubebuilder and operator-sdk operators are built. Similar to 1), if there are library bits that are applicable to all Go operators, we should consider adding them to controller-runtime instead of operator-sdk.

I think we can collect the feature additions relevant to SDK here for now. Maybe just keep a running checklist in the description of features we want to consider for inclusion?

OK, should I add my findings as comments here? Note, however, that I'm the author of one of these libraries so I am not without bias :)

Two other important points that are relevant here:

1. We're working on integrating with Kubebuilder, which will result in `operator-sdk` using Kubebuilder project layouts for Go operators. If any of the features are related to project files or project layout AND they would be applicable to ALL Go operators, we should consider adding the features to Kubebuilder instead of operator-sdk

2. We also heavily use `controller-runtime`, which is the library on which kubebuilder and operator-sdk operators are built. Similar to 1), if there are library bits that are applicable to all Go operators, we should consider adding them to controller-runtime instead of operator-sdk.

OK, good to know. Wasn't aware of these efforts and have to admit that I'm not familiar with kubebuilder at all… but it does indeed make sense to include the functionality at the right level.

@luksa fyi

@metacosm I think we list out all possible features that are candidates for inclusion in SDK (or upstream) and then we can triage to determine:

1. if the feature already exists
2. whether the feature is general enough to be included in the SDK (or upstream)
3. if 2., which specific repo is best suited to accept the feature.

Comments are fine and perhaps we can keep a running task list in the main issue description to make it easier to see the proposed features at a glance.

We need a framework to assess non-core/immature but promising contributions to operator-framework repos. To me, non-core means something not every operator author in all stages of operator development would benefit from. Maturity comes from community support for and activity in a particular project. A great example of both cases is Composable, which is relevant for advanced use cases but is still immature. There were two suggestions made on how to get such contributions into operator-framework:

1. Have a contrib/ directory in a repo. Contributions would get their own subdirectory containing a submodule and OWNERS/MAINTAINERS.
2. Have a separate repo ex. operator-framework/operator-tools-incubator. Contributions would get their own subdirectory containing a submodule and OWNERS/MAINTAINERS.

Submodules separate dependency concerns between projects, a necessity when dealing with k8s dependencies. Delegating ownership of each submodule to the contributors scales maintenance responsibilities so maintenance is not gate-kept by operator-framework maintainers. A graduation criteria, which still needs to be discussed, needs to be established such that projects that pass criteria will either get their own operator-framework repo or a place in a relevant existing repo.

The process I'd like to see for contributions of external code:

1. Proposal PR to the enhancements repo.
   1. The proposal must either describe which repo should receive the contribution and why, or argue why the contribution should have a standalone repo.
2. Once the proposal PR is merged, an implementation (potentially preceded by a design doc) PR should be made against the relevant repo.
   1. The implementation PR must contain a promise of ownership in the form of an OWNERS/MAINTAINERS file, AND at least one sponsor who currently maintains a stable (core) operator-framework project. These sponsors will be operator-framework points of contact for the community involved with the contribution.
3. Once the implementation PR is merged, the graduation process beings.

Upstream k8s probably has a more fleshed-out contribution process, which is probably better than the above one.

@dmesser @robszumski @joelanford @metacosm thoughts?

Hi @estroz,

My opinion over the above suggestions is: If we add a dir in the repo with code implementations there then:

• It started to be part of the project which means that requires to be maintained and supported
• contributor began to need to dealing with anyway
• more dependencies will be added to the project, which will increase the complexity to keep it supported and maintained.
• Also, what real advantages for users the above suggestions will bring? Would make sense have a magic box with many solutions not officially "supported", "maintained" or documented? Will users able to use it?

And then, If we added a repo in the org, it also started to be part of the org responsibilities and promoted and "blessed" by the framework.

So, I do not think that we should move with it.

What we could do anyway?

May we just add a doc in SDK with links to their projects README and then, in this way, allows contributors to share their helpers and tools which are NOT supported or maintained by the org.

What should be added or not in SDK?

IHMO: We should procedure as it is done in the other projects:

Is the suggestion valuable, relevant, generic and mature enough to be added in the project?

• Yes (👍): great news 🥇 We add/merge and start to support, doc and maintain it.
• No (👎): then we clarify the gaps and why not that cannot be added yet in the project and allow contributors working on to improve their ideas to achieve the end goal to make that part of SDK.

• Having explicit ownership through an OWNERS file and guidelines on contribution/graduation helps prevent this, although bugs/features will be filed against contributed projects that core maintainers will have to at least look at from time to time, which is a good thing IMO. I don't foresee too much overhead.
• How do you mean?
• With multi-module repositories we can create independently managed subdirs in a repo.
• These contributions will still go through a proposal process. If the community decides to bring a contribution into operator-framework, they clearly want to use that project. Of course we need to set guidelines as to what maintainership looks like, removing dead contributions, etc.

Ultimately we want to improve an operator developer's experience with better tooling/libraries/documentation. If someone/some group has a contribution that improves upon the development experience, but might be immature or only applies to certain situations (but is still useful), we neither want to turn them away nor immediately maintain the contribution. The best middle ground is to let the contribution's owners maintain the project under "incubating" status somewhere in operator-framework. The practice of incubation is commonly accepted across framework-style/focussed technology groups, ex. CNCF.

Do I get this talk on governance right to mean that the current thinking is to not include any of the functionality described above at this point?

@metacosm not at all. I’m suggesting where to put it and how to manage it.

HI @estroz,

IMO: The complexities point out are NOT just about the modules/deps. In a way or another, we will start to need to handle and work with any source code added on the project.

Also, I agree that it is valid for who made the collab. However, if it is not generic, mature, document enough would it attend others? Will it not just address very specific scenarios? What will be the criteria to add allow merge code in this dir?

In this way, I think would be valid we came back and think about:

• What are we trying to achieve by applying your suggestions A and B?
• What are we trying to solving with the above suggestions?

Are we not just trying to give visibility to these helpers and tools? Are we trying to allow collaboration on them? I mean let people know that they exist and may collab with them. If yes, why not address this a doc?

@estroz Thanks for starting this initiative. Couple of questions:

• How do we plan to integrate potentially immature contributions with the mature code base while providing a clear demarkation line for users (either in docs, beta subcommands or comments in scaffolded code)?
• What about contributions that extend existing functionality rather than adding net-new?

Sounds almost like we need a plugin system :)

My point of view on this issue is that it's meant to determine which functionality (if any) from the above libraries / framework would be good candidate for inclusion with the operator-sdk code base, i.e. functionality that is deemed useful for any operator developer.

If none of the provided functionality is deemed interesting for inclusion within the sdk code base, then I don't think there's a point apart from maybe documenting the existence of such libraries pointing users to them and letting them decide whether or not they want to use such facilities. The whole idea was to improve the sdk itself, not figure out how third parties could extend it.

Has anyone looked at the provided functionality? A document has been created to summarize the concepts of each library: https://docs.google.com/document/d/1NlbVfSi84LkebEhWGWgRcTXGghF0CQaHyIQjA6i-7G4/edit. Note that there are without doubt other libraries providing similar functionality, which to me, is a strong indicator that at least some of these features should be built-in in the sdk.

It's on my plate to triage the doc. Thanks SO much to the folks who helped put that together! :pray:

As I mentioned before, my current plan is to make several buckets:

1. upstream to kubebuilder (scaffolding related improvements)
2. upstream to controller-runtime (controller helpers that are widely applicable to all Go operators)
3. upstream to operator-sdk (controller helpers that are applicable to the set of operators that are in-scope for operator-sdk, but not all Go operators)
4. add to new operator-framework repo (experimental, niche, or immature features, or anything that has some question about its longevity and maintainability)
5. add to new (or existing?) openshift repo (Openshift-specific features and helpers)

Sorry for the radio silence so far. There's just lots to do, and not enough time to do it!

@joelanford that sounds like a good delineation.

@dmesser for scaffolded code or packages, a contrib folder/repo for plugins sounds like a good idea. Having new contrib commands under alpha/beta sgtm, although we might want to provide an "experimental" release of the SDK that includes these contrib commands. For extensions of current SDK packages used in an operator project, use at your own risk by changing imports and reading contrib documentation should be as far as the core SDK goes to support contrib usage.

@metacosm

functionality that is deemed useful for any operator developer.

Having an in-or-out approach seems too restrictive since different operators have different capability levels; more broadly, there is no one way to do all things. Also given the rapidly developing ecosystem, the SDK should strive to be inclusive rather than exclusive. We can always reject a proposal, or remove an unused contrib; it is much harder to deprecate/remove core functionality when something better (not talking about the next shiny thing) comes along. However the SDK strives to be opinionated about how operators are developed, which I think is a reasonable argument against adding contributions as I've outlined them, and for an in-or-out approach as you suggest

And can you clarify

The whole idea was to improve the sdk itself, not figure out how third parties could extend it.

What kind of extension are you talking about?

@joelanford that sounds like a good delineation.

Based on this typography, it seems as if the library contributions would be best targeted at controller-runtime rather than this project.

@metacosm

functionality that is deemed useful for any operator developer.

Having an in-or-out approach seems too restrictive since different operators have different capability levels; more broadly, there is no one way to do all things. Also given the rapidly developing ecosystem, the SDK should strive to be inclusive rather than exclusive. We can always reject a proposal, or remove an unused contrib; it is much harder to deprecate/remove core functionality when something better (not talking about the next shiny thing) comes along. However the SDK strives to be opinionated about how operators are developed, which I think is a reasonable argument against adding contributions as I've outlined them, and for an in-or-out approach as you suggest

This is not what I'm suggesting but there are sufficient commonalities in these libraries and probably others to maybe warrant inclusion in one of the projects… Have you taken a look at the linked projects?

And can you clarify

The whole idea was to improve the sdk itself, not figure out how third parties could extend it.

What kind of extension are you talking about?

Not any in particular. Seems just that the discussion so far has been centered around governance as opposed to discussing the merits of the functionality provided by the linked libraries and whether or not they fit in this project. Based on @joelanford's comment and my own analysis, it seems controller-runtime might be more suited.

Seems just that the discussion so far has been centered around governance as opposed to discussing the merits of the functionality provided by the linked libraries and whether or not they fit in this project.

Yeah sorry I derailed pretty hard here.

Based on @joelanford's comment and my own analysis, it seems controller-runtime might be more suited.

Agreed, it looks like at least APIs like CreateOrUpdate would be suited for controller-runtime.

Hello, any news on this thread? I feel that the composable-sdk is falling through the cracks...

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten /remove-lifecycle stale

/remove-lifecycle rotten

/lifecycle frozen

Another library recently brought to my attention: https://github.com/darkowlzz/operator-toolkit.

@estroz @joelanford @camilamacedo86 Hello all, I wonder what is the status of this issue? The Composable Operator (https://github.com/IBM/composable) now has a handful of users (@xvzf). Is there still an interest in integration with Operator-SDK?

kubebuilder deploy-image plugin is doing this https://book.kubebuilder.io/plugins/deploy-image-plugin-v1-alpha.html

If any other features should be added to SDK, lets create a new issue.