jetstack-secure
jetstack-secure copied to clipboard
Generate RBAC based on the current config
Generate RBAC based on the current config, and in such a way that the code can be shared with the backend.
More context on this one.
platform.jetstack.io has a wizard that guides the user to connect a cluster to the platform.
It generates a command for the user to run like this one:
$ { kubectl apply -f - <<EOF
apiVersion: v1
kind: Namespace
metadata:
name: jetstack-secure
---
apiVersion: v1
kind: ConfigMap
metadata:
name: agent-config
namespace: jetstack-secure
data:
config.yaml: |-
server: "https://platform.jetstack.io"
organization_id: "my-organization"
cluster_id: "my-cluster"
data-gatherers:
# gather k8s apiserver version information
- kind: "k8s-discovery"
name: "k8s-discovery"
# pods data is used in the pods and application_versions packages
- kind: "k8s-dynamic"
name: "k8s/pods"
config:
resource-type:
resource: pods
version: v1
# gather services for pod readiness probe rules
- kind: "k8s-dynamic"
name: "k8s/services"
config:
resource-type:
resource: services
version: v1
# gather higher level resources to ensure data to determine ownership is present
- kind: "k8s-dynamic"
name: "k8s/deployments"
config:
resource-type:
version: v1
resource: deployments
group: apps
- kind: "k8s-dynamic"
name: "k8s/replicasets"
config:
resource-type:
version: v1
resource: replicasets
group: apps
- kind: "k8s-dynamic"
name: "k8s/statefulsets"
config:
resource-type:
version: v1
resource: statefulsets
group: apps
- kind: "k8s-dynamic"
name: "k8s/daemonsets"
config:
resource-type:
version: v1
resource: daemonsets
group: apps
- kind: "k8s-dynamic"
name: "k8s/jobs"
config:
resource-type:
version: v1
resource: jobs
group: batch
- kind: "k8s-dynamic"
name: "k8s/cronjobs"
config:
resource-type:
version: v1beta1
resource: cronjobs
group: batch
# gather resources for cert-manager package
- kind: "k8s-dynamic"
name: "k8s/secrets"
config:
resource-type:
version: v1
resource: secrets
- kind: "k8s-dynamic"
name: "k8s/certificates"
config:
resource-type:
group: cert-manager.io
version: v1
resource: certificates
- kind: "k8s-dynamic"
name: "k8s/ingresses"
config:
resource-type:
group: networking.k8s.io
version: v1
resource: ingresses
- kind: "k8s-dynamic"
name: "k8s/certificaterequests"
config:
resource-type:
group: cert-manager.io
version: v1
resource: certificaterequests
- kind: "k8s-dynamic"
name: "k8s/issuers"
config:
resource-type:
group: cert-manager.io
version: v1
resource: issuers
- kind: "k8s-dynamic"
name: "k8s/clusterissuers"
config:
resource-type:
group: cert-manager.io
version: v1
resource: clusterissuers
- kind: "k8s-dynamic"
name: "k8s/googlecasissuers"
config:
resource-type:
group: cas-issuer.jetstack.io
version: v1beta1
resource: googlecasissuers
- kind: "k8s-dynamic"
name: "k8s/googlecasclusterissuers"
config:
resource-type:
group: cas-issuer.jetstack.io
version: v1beta1
resource: googlecasclusterissuers
- kind: "k8s-dynamic"
name: "k8s/awspcaissuer"
config:
resource-type:
group: awspca.cert-manager.io
version: v1beta1
resource: awspcaissuers
- kind: "k8s-dynamic"
name: "k8s/awspcaclusterissuers"
config:
resource-type:
group: awspca.cert-manager.io
version: v1beta1
resource: awspcaclusterissuers
- kind: "k8s-dynamic"
name: "k8s/mutatingwebhookconfigurations"
config:
resource-type:
group: admissionregistration.k8s.io
version: v1
resource: mutatingwebhookconfigurations
- kind: "k8s-dynamic"
name: "k8s/validatingwebhookconfigurations"
config:
resource-type:
group: admissionregistration.k8s.io
version: v1
resource: validatingwebhookconfigurations
---
apiVersion: v1
kind: Secret
metadata:
name: agent-credentials
namespace: jetstack-secure
data:
credentials.json: <REDACTED>
EOF
} && kubectl apply -f https://platform.jetstack.io/agent_installation.yaml
That long command is actually two commands concatenated. The first one applies this yaml with kubectl:
apiVersion: v1
kind: Namespace
metadata:
name: jetstack-secure
---
apiVersion: v1
kind: ConfigMap
metadata:
name: agent-config
namespace: jetstack-secure
data:
config.yaml: |-
server: "https://platform.jetstack.io"
organization_id: "my-organization"
cluster_id: "my-cluster"
data-gatherers:
# gather k8s apiserver version information
- kind: "k8s-discovery"
name: "k8s-discovery"
# pods data is used in the pods and application_versions packages
- kind: "k8s-dynamic"
name: "k8s/pods"
config:
resource-type:
resource: pods
version: v1
# gather services for pod readiness probe rules
- kind: "k8s-dynamic"
name: "k8s/services"
config:
resource-type:
resource: services
version: v1
# gather higher level resources to ensure data to determine ownership is present
- kind: "k8s-dynamic"
name: "k8s/deployments"
config:
resource-type:
version: v1
resource: deployments
group: apps
- kind: "k8s-dynamic"
name: "k8s/replicasets"
config:
resource-type:
version: v1
resource: replicasets
group: apps
- kind: "k8s-dynamic"
name: "k8s/statefulsets"
config:
resource-type:
version: v1
resource: statefulsets
group: apps
- kind: "k8s-dynamic"
name: "k8s/daemonsets"
config:
resource-type:
version: v1
resource: daemonsets
group: apps
- kind: "k8s-dynamic"
name: "k8s/jobs"
config:
resource-type:
version: v1
resource: jobs
group: batch
- kind: "k8s-dynamic"
name: "k8s/cronjobs"
config:
resource-type:
version: v1beta1
resource: cronjobs
group: batch
# gather resources for cert-manager package
- kind: "k8s-dynamic"
name: "k8s/secrets"
config:
resource-type:
version: v1
resource: secrets
- kind: "k8s-dynamic"
name: "k8s/certificates"
config:
resource-type:
group: cert-manager.io
version: v1
resource: certificates
- kind: "k8s-dynamic"
name: "k8s/ingresses"
config:
resource-type:
group: networking.k8s.io
version: v1
resource: ingresses
- kind: "k8s-dynamic"
name: "k8s/certificaterequests"
config:
resource-type:
group: cert-manager.io
version: v1
resource: certificaterequests
- kind: "k8s-dynamic"
name: "k8s/issuers"
config:
resource-type:
group: cert-manager.io
version: v1
resource: issuers
- kind: "k8s-dynamic"
name: "k8s/clusterissuers"
config:
resource-type:
group: cert-manager.io
version: v1
resource: clusterissuers
- kind: "k8s-dynamic"
name: "k8s/googlecasissuers"
config:
resource-type:
group: cas-issuer.jetstack.io
version: v1beta1
resource: googlecasissuers
- kind: "k8s-dynamic"
name: "k8s/googlecasclusterissuers"
config:
resource-type:
group: cas-issuer.jetstack.io
version: v1beta1
resource: googlecasclusterissuers
- kind: "k8s-dynamic"
name: "k8s/awspcaissuer"
config:
resource-type:
group: awspca.cert-manager.io
version: v1beta1
resource: awspcaissuers
- kind: "k8s-dynamic"
name: "k8s/awspcaclusterissuers"
config:
resource-type:
group: awspca.cert-manager.io
version: v1beta1
resource: awspcaclusterissuers
- kind: "k8s-dynamic"
name: "k8s/mutatingwebhookconfigurations"
config:
resource-type:
group: admissionregistration.k8s.io
version: v1
resource: mutatingwebhookconfigurations
- kind: "k8s-dynamic"
name: "k8s/validatingwebhookconfigurations"
config:
resource-type:
group: admissionregistration.k8s.io
version: v1
resource: validatingwebhookconfigurations
---
apiVersion: v1
kind: Secret
metadata:
name: agent-credentials
namespace: jetstack-secure
data:
credentials.json: <REDACTED>
We can see there 3 different yaml objects. A Namespace
, a ConfigMap
and a Secret
.
The ConfigMap
embeds config.yaml
, which is the configuration file for the agent.
Now, let's have a look at the second command:
kubectl apply -f https://platform.jetstack.io/agent_installation.yaml
It is applying another yaml. If we inspect the content (e.g. by doing curl -L https://platform.jetstack.io/agent_installation.yaml
) we will see something like:
apiVersion: v1
kind: Namespace
metadata:
name: jetstack-secure
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: agent
namespace: jetstack-secure
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: jetstack-secure-agent-node-reader
rules:
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: jetstack-secure-agent-secret-reader
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: jetstack-secure-agent-cert-manager-reader
rules:
- apiGroups: ["cert-manager.io"]
resources:
- certificates
- certificaterequests
- issuers
- clusterissuers
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: jetstack-secure-agent-googlecas-reader
rules:
- apiGroups: ["cas-issuer.jetstack.io"]
resources:
- googlecasissuers
- googlecasclusterissuers
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: jetstack-secure-agent-awspca-reader
rules:
- apiGroups: ["awspca.cert-manager.io"]
resources:
- awspcaissuers
- awspcaclusterissuers
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: jetstack-secure-agent-get-webhooks
rules:
- apiGroups: ["admissionregistration.k8s.io"]
resources:
- validatingwebhookconfigurations
- mutatingwebhookconfigurations
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: jetstack-secure-agent-secret-reader
roleRef:
kind: ClusterRole
name: jetstack-secure-agent-secret-reader
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: agent
namespace: jetstack-secure
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: jetstack-secure-agent-cert-manager-reader
roleRef:
kind: ClusterRole
name: jetstack-secure-agent-cert-manager-reader
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: agent
namespace: jetstack-secure
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: jetstack-secure-agent-googlecas-reader
roleRef:
kind: ClusterRole
name: jetstack-secure-agent-googlecas-reader
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: agent
namespace: jetstack-secure
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: jetstack-secure-agent-awspca-reader
roleRef:
kind: ClusterRole
name: jetstack-secure-agent-awspca-reader
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: agent
namespace: jetstack-secure
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: jetstack-secure-agent-get-webhooks
roleRef:
kind: ClusterRole
name: jetstack-secure-agent-get-webhooks
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: agent
namespace: jetstack-secure
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: jetstack-secure-agent-node-reader
roleRef:
kind: ClusterRole
name: jetstack-secure-agent-node-reader
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: agent
namespace: jetstack-secure
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: jetstack-secure-agent-cluster-viewer
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: view
subjects:
- kind: ServiceAccount
name: agent
namespace: jetstack-secure
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: agent
namespace: jetstack-secure
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: agent
template:
metadata:
labels:
app.kubernetes.io/name: agent
spec:
serviceAccountName: agent
volumes:
- name: config
configMap:
name: agent-config
- name: credentials
secret:
secretName: agent-credentials
containers:
- name: agent
image: quay.io/jetstack/preflight:v0.1.31
args:
- "agent"
- "-c"
- "/etc/jetstack-secure/agent/config/config.yaml"
- "-k"
- "/etc/jetstack-secure/agent/credentials/credentials.json"
- "-p"
- "0h1m0s"
volumeMounts:
- name: config
mountPath: "/etc/jetstack-secure/agent/config"
readOnly: true
- name: credentials
mountPath: "/etc/jetstack-secure/agent/credentials"
readOnly: true
resources:
requests:
memory: "200Mi"
cpu: "200m"
limits:
memory: "200Mi"
cpu: "200m"
This last yaml contains the actual workload for the agent (a Deployment
) and also the RBAC resources to let the agent gather data from the cluster.
This RBAC configuration needs to be aligned with the datagatherers configured in the agent configuration (remember config.yaml
).
What we are trying to achieve here is to create a generator for these two things: default datagatherers configuration and default RBAC.
I've been working a little with Wenlin on this today and my plan was, if there was more time, to build this incrementally with TDD in stages, e.g.
- GVRs in, RBAC string templated out
- GVRs in, RBAC structs out
- GVRs and Namespaces in, scoped rbac generated based on some list of known namespaces in the case of excluded namespaces
- means of getting GVRs from loaded dynamic data gatherers.
- adding a command to spit this all out.
- profit
- GVRs and Namespaces in, scoped rbac generated based on some list of known namespaces in the case of excluded namespaces
I think this does not make much sense. At generation time is not possible to know all the existing namespaces so you can filter out the exclusion list.
@charlieegan3 I am not sure I follow this.
You are correct, I was imagining that it'd be possible to supply a list of namespaces that comes from kubectl get ns if the agent were running locally using the installers credentials.
I'm thinking that the first use of this is going to be a command in the agent, and if the agent, running in that command had access to the installers creds, it could get a list of all namespaces.
We might not be able to do this when using it from the platform side, however I think it's a feature worth adding here in the agent.
The RBAC generation command has been added to the agent CLI command in #258 we are going to continue the work, new default config generation, In following PRs.