operator-framework
New feature: list-operatorgroups
This is a new feature: kubectl-operator list-operatorgroups
The goal here is to:
- List out all OperatorGroup and related namespaces as Tenants
- Provide both Tabular and Mermaid support
Example output using a set of test data found in assets/operatorgroup-list-test:
$ bin/kubectl-operator list-operatorgroups -A
TENANT TYPE SUBTENANTS TARGETNAMESPACES PARENTTENANTS
oa OwnNamespace oa
oaa SingleNamespace oaa1 oaa1
oaa1 OwnNamespace oaa1 oaa,obb
ob MultiNamespace ob1 ob,ob1
ob1 OwnNamespace ob1 ob
obb SingleNamespace oaa1 oaa1
oc MultiNamespace oc1,oc2 oc1,oc2
oc1 OwnNamespace oc1 oc
oc2 OwnNamespace oc2 oc
occ SingleNamespace occ1 occ1
occ1 SingleNamespace occ11 occ11 occ
occ11 OwnNamespace occ11 occ1,odd2
od MultiNamespace od1,od2 od,od1,od2
od1 MultiNamespace od11,od12 od1,od11,od12 od
od11 OwnNamespace od11 od1
od12 OwnNamespace od12 od1
od2 MultiNamespace od21,od22 od2,od21,od22 od
od21 OwnNamespace od21 od2
od22 OwnNamespace od22 od2
odd SingleNamespace odd1 odd1
odd1 SingleNamespace odd11 odd11 odd
odd2 SingleNamespace occ11 occ11
oe SingleNamespace ne1 ne1
of MultiNamespace nf1,nf2 nf1,nf2
og MultiNamespace og1,og2 og1,og2
og1 MultiNamespace ng11,ng12 ng11,ng12 og
og2 MultiNamespace ng21,ng22 ng21,ng22 og
oh MultiNamespace oh1,th11 oh1,th11
oh1 SingleNamespace nh11 nh11 oh
openshift-monitoring AllNamespaces
openshift-operator-lifecycle-manager OwnNamespace openshift-operator-lifecycle-manager
openshift-operators AllNamespaces
$ bin/kubectl-operator list-operatorgroups -A -g
flowchart LR
classDef tenant fill:#eFe,stroke:#000;
classDef bad fill:#f99,stroke:#000;
classDef ns fill:#fff,stroke:#000,stroke-width:1px;
classDef im-OwnNamespace fill:#eef,stroke:#000,stroke-width:2px;
classDef im-SingleNamespace fill:#ddf,stroke:#000,stroke-width:2px;
classDef im-MultiNamespace fill:#ccf,stroke:#000,stroke-width:2px;
subgraph Legend [Legend]
direction TB LNS[Namespace]:::ns
LOGOWN(OwnNamespace):::im-OwnNamespace
LOGSINGLE(SingleNamespace):::im-SingleNamespace
LOGMULTI(MultiNamespace):::im-MultiNamespace
LOGNS[Operand Namespace]:::ns
end t-multicluster-engine:::tenant
subgraph t-multicluster-engine[multicluster-engine]
direction LR
multicluster-engine(multicluster-engine):::im-OwnNamespace --> multicluster-engine(multicluster-engine):::im-OwnNamespace
end
t-oc:::tenant
subgraph t-oc[oc]
direction LR
oc(oc):::im-MultiNamespace --> oc1(oc1):::im-OwnNamespace
oc1(oc1):::im-OwnNamespace --> oc1(oc1):::im-OwnNamespace
oc(oc):::im-MultiNamespace --> oc2(oc2):::im-OwnNamespace
oc2(oc2):::im-OwnNamespace --> oc2(oc2):::im-OwnNamespace
end
t-obb:::tenant
subgraph t-obb[obb]
direction LR
obb(obb):::im-SingleNamespace --> oaa1(oaa1):::bad
oaa1(oaa1):::bad --> oaa1(oaa1):::bad
end
t-oe:::tenant
subgraph t-oe[oe]
direction LR
oe(oe):::im-SingleNamespace --> ne1[ne1]:::ns
end
t-occ:::tenant
subgraph t-occ[occ]
direction LR
occ(occ):::im-SingleNamespace --> occ1(occ1):::im-SingleNamespace
occ1(occ1):::im-SingleNamespace --> occ11(occ11):::bad
occ11(occ11):::bad --> occ11(occ11):::bad
end
t-odd:::tenant
subgraph t-odd[odd]
direction LR
odd(odd):::im-SingleNamespace --> odd1(odd1):::im-SingleNamespace
odd1(odd1):::im-SingleNamespace --> odd11[odd11]:::ns
end
t-og:::tenant
subgraph t-og[og]
direction LR
og(og):::im-MultiNamespace --> og1(og1):::im-MultiNamespace
og1(og1):::im-MultiNamespace --> ng11[ng11]:::ns
og1(og1):::im-MultiNamespace --> ng12[ng12]:::ns
og(og):::im-MultiNamespace --> og2(og2):::im-MultiNamespace
og2(og2):::im-MultiNamespace --> ng21[ng21]:::ns
og2(og2):::im-MultiNamespace --> ng22[ng22]:::ns
end
t-ob:::tenant
subgraph t-ob[ob]
direction LR
ob(ob):::im-MultiNamespace --> ob(ob):::im-MultiNamespace
ob(ob):::im-MultiNamespace --> ob1(ob1):::im-OwnNamespace
ob1(ob1):::im-OwnNamespace --> ob1(ob1):::im-OwnNamespace
end
t-of:::tenant
subgraph t-of[of]
direction LR
of(of):::im-MultiNamespace --> nf1[nf1]:::ns
of(of):::im-MultiNamespace --> nf2[nf2]:::ns
end
t-oh:::tenant
subgraph t-oh[oh]
direction LR
oh(oh):::im-MultiNamespace --> oh1(oh1):::im-SingleNamespace
oh1(oh1):::im-SingleNamespace --> nh11[nh11]:::ns
oh(oh):::im-MultiNamespace --> th11[th11]:::ns
end
t-oaa:::tenant
subgraph t-oaa[oaa]
direction LR
oaa(oaa):::im-SingleNamespace --> oaa1(oaa1):::bad
end
t-odd2:::tenant
subgraph t-odd2[odd2]
direction LR
odd2(odd2):::im-SingleNamespace --> occ11(occ11):::bad
end
t-oa:::tenant
subgraph t-oa[oa]
direction LR
oa(oa):::im-OwnNamespace --> oa(oa):::im-OwnNamespace
end
t-od:::tenant
subgraph t-od[od]
direction LR
od(od):::im-MultiNamespace --> od(od):::im-MultiNamespace
od(od):::im-MultiNamespace --> od1(od1):::im-MultiNamespace
od1(od1):::im-MultiNamespace --> od1(od1):::im-MultiNamespace
od1(od1):::im-MultiNamespace --> od11(od11):::im-OwnNamespace
od11(od11):::im-OwnNamespace --> od11(od11):::im-OwnNamespace
od1(od1):::im-MultiNamespace --> od12(od12):::im-OwnNamespace
od12(od12):::im-OwnNamespace --> od12(od12):::im-OwnNamespace
od(od):::im-MultiNamespace --> od2(od2):::im-MultiNamespace
od2(od2):::im-MultiNamespace --> od2(od2):::im-MultiNamespace
od2(od2):::im-MultiNamespace --> od21(od21):::im-OwnNamespace
od21(od21):::im-OwnNamespace --> od21(od21):::im-OwnNamespace
od2(od2):::im-MultiNamespace --> od22(od22):::im-OwnNamespace
od22(od22):::im-OwnNamespace --> od22(od22):::im-OwnNamespace
end
t-openshift-operator-lifecycle-manager:::tenant
subgraph t-openshift-operator-lifecycle-manager[openshift-operator-lifecycle-manager]
direction LR
openshift-operator-lifecycle-manager(openshift-operator-lifecycle-manager):::im-OwnNamespace --> openshift-operator-lifecycle-manager(openshift-operator-lifecycle-manager):::im-OwnNamespace
end
[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: cdjohnson
Once this PR has been reviewed and has the lgtm label, please assign joelanford for approval by writing /assign @joelanford in a comment. For more information see the Kubernetes Code Review Process.
The full list of commands accepted by this bot can be found here.
Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment
![openshift-ci[bot] avatar](https://avatars.githubusercontent.com/in/91280?v=4 "Published by a pub.dev verified publisher"){kind=small}
Curious, what use case does this address?
Our larger customers want to create large deployment topologies within a single cluster that allow different isolation layers to allow different instances of Products (Operators and Operands). Our products are made up of many operators, some of which are shared and some are not, so it can get fairly complicated quickly. In the most complicated scenario, you may have a multiple Trees of Namespaces, where each layer in the tree represents a different Control Plane, or Isolation Layer.
The goal of this was to help developers and SREs visualize these more complex deployment scenarios and validate their configurations.
Wouldn't this question be best answered from the perspective of an operator configured / scoped via an OperatorGroup ? I.e. instead of listing the OperatorGroup themselves, which are sort of an abstract concept rather, we could list all operators and their scope
@dmesser This is in the scope of the current OLM APIs and it's current capabilities. This is showing what the Deployment Environment/Topology that the Operators will be installed into. Although the OperatorGroup is an abstraction, customers need to set them up correctly and orchestrate the operators to install into that environment to have success.
So, this shows that the Namespaces are setup properly even before the first operator is provisioned.
So:
- The customer needs to have awareness of Namespaces and OperatorGroups to build the right topologies (new tool could help here).
- Validate that they are setup properly (this tool).
- Deploy the operators and operands into those namespaces.
- Validate that the Operators are installed properly (new tool....?)
I guess I am trying to understand the situation in which a user would use this command. It's not obvious to me.
Let's say I want to install an operator or a set of related operators. Why is the question which operator groups already exist interesting to me? Wouldn't I be rather interest in:
- discovering which operators are available for me to install (this plugin can do that)
- select the version of the operator I want to install (this plugin cannot do this yet, but we have made in-roads into this with
packageserver) - installing the operator(s) which a specific scope (this tool can do this)
- validate the operators are installed and healthy (this tool will check the install, install health needs more discussion)
I don't see OperatorGroups in here as a primary concern. They are an implementation detail of how the operator got configured and installed. The plugin sets them up transparently in step 2. If there are 2 or 10 OperatorGroups installed in other namespaces, potentially scoped the (partially) same way the next one is about to come down, doesn't really matter IMHO. What matters is that we notice when an operator has already be installed and configured to be available in the same namespace / namespaces you are targeting. I am not sure that we have made that yet.
The problem is that we can't rely on OLM to resolve and provision the dependent operators for us, because they live at different scopes. Some customers (big ones) want to use the principle of least privilege a the Namespace layer. This is an extremely "advanced" use case, but one that we need to support.
Primary goals:
- No Cluster permissions.
- Namespace isolation (RBAC, Network, Quota)
- Principle of Least Privilege (nesting doll/concentric circle approach)
Let's say we have 4 layers:
- Cluster (Certificate Management, License Management)
- Watches All or Many namespaces. These are cluster-singletons.
- Shared Tenant Operators (IAM and other shared services. Could represent a Company)
- IAM CRs are likely created here... a shared UI, User Registry, etc that's common to all product instances
- Tenant Instance Operators (Could represent a Department)
- Application oriented operators and middleware.
- Has Kube RBAC
- Tenant Operands (CRs only)
- The instances of the Application. Or a "Data Plane"
- No Kube RBAC at all
In order to realize this, we have a tree of namespaces and nested operator groups. 1 manages 2 manages 3 manages 4
Because of this, we need to design the Namespaces and their OperatorGroups ahead of time, essentially Modeling the desired topology, and then deploy the Operators and Operands into that topology, not depending on OLM to auto-provision dependencies.
Here's a picture:
@dmesser So, I think I can clarify what the benefit of this tool could be:
- As a Cluster Admin, how can I explore how the current Tenancy Topology is configured, so I can determine which Namespace to install my Operator into?
- How can I detect mis-configurations of operator groups which might result in Operator deployment problems?
@cdjohnson: PR needs rebase.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
