Prevent kubequery generated configurations from containing secrets

[nonpunctual]

Kubequery has get on secrets - "a super powerful role."

A query on secrets does not actually return the secret, but annotations are returned.

kubectl apply creates an annotation called kubectl.kubernetes.io/last-applied-configuration documented here:

http://kubectl.kubernetes.io/last-applied-configuration  https://kubernetes.io/docs/tasks/manage-kubernetes-objects/declarative-config/

which will contain secrets. Customer says no access to secrets is necessary for building these configurations.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kubequery-clusterrole
  labels:
    app.kubernetes.io/name: kubequery-clusterrole
    app.kubernetes.io/part-of: kubequery
    app.kubernetes.io/version: latest
rules:
- apiGroups: [""]
  verbs: ["get", "list", "watch"]
  resources:
    - bindings
    - componentstatuses
    - configmaps
    - endpoints
    - events
    - limitranges
    - namespaces
    - nodes
    - persistentvolumeclaims
    - persistentvolumes
    - pods
    - podtemplates
    - replicationcontrollers
    - resourcequotas
    - serviceaccounts
    - services
- apiGroups: ["admissionregistration.k8s.io", "apps", "autoscaling", "batch", "events.k8s.io", "networking.k8s.io", "policy", "rbac.authorization.k8s.io", "storage.k8s.io"]
  resources: ["*"]
  verbs: ["get", "list", "watch"]

See: https://github.com/fleetdm/fleet/blob/main/tools/kubequery/kubequery-fleet.yml#L41-L44

- apiGroups: ["", "admissionregistration.k8s.io", "apps", "autoscaling", "batch", "events.k8s.io", "networking.k8s.io", "policy", "rbac.authorization.k8s.io", "storage.k8s.io"]
  resources: ["*"]
  verbs: ["get", "list", "watch"]

The 1st entry in apiGroups is "", which is the api group of fundamental k8s resources. The "*" in resources: includes secrets. Removing the cluster role above was to remove "" from the list, which specifies the resources and leaves out secrets.

Problem

Secrets should be suppressed in output & prevented from writing to version control systems or out to configuration files so they are not persistently accessible.

Potential solutions

1. Prevent secrets from being written to kubequery configurations
2. Exclude annotation kubectl.kubernetes.io/last-applied-configuration

[yoderme]

I would think that most people deploying production k8s would be by far the most comfortable if this tool didn't have access to secrets in the first place. Fortunately this is easy to control by changing the ClusterRole - but you also have to know what you're doing.

[noahtalerman]

Hey @nonpunctual and @yoderme!

Kubequery has get on secrets - "a super powerful role."

The 1st entry in apiGroups is "", which is the api group of fundamental k8s resources. The "*" in resources: includes secrets. Removing the cluster role above was to remove "" from the list, which specifies the resources and leaves out secrets.

I'm not sure I'm following...

How does one modify the kubequery-fleet.yml file to remove the ability to get secrets?

Does one remove the resources: ["*"] from the ClusterRole?

For example:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kubequery-clusterrole
  labels:
    app.kubernetes.io/name: kubequery-clusterrole
    app.kubernetes.io/part-of: kubequery
    app.kubernetes.io/version: latest
rules:
- apiGroups: ["", "admissionregistration.k8s.io", "apps", "autoscaling", "batch", "events.k8s.io", "networking.k8s.io", "policy", "rbac.authorization.k8s.io", "storage.k8s.io"]
  # resources: ["*"]
  verbs: ["get", "list", "watch"]

[nonpunctual]

Solution 2 is what the customer has done to workaround this. If that were the default it would be a solution.

Preferable would be no ability for secrets to go to stdout at all during the creation of these configs.

[noahtalerman]

Solution 2 is what the customer has done to workaround this.

2. Exclude annotation kubectl.kubernetes.io/last-applied-configuration

I still don't follow. That's^ a broken link?

Preferable would be no ability for secrets to go to stdout at all during the creation of these configs.

What would I do in Fleet's config file to achieve this?

[yoderme]

I still don't follow. That's^ a broken link?

Oh, that's not a broken link, that's the name of a label. See https://kubernetes.io/docs/tasks/manage-kubernetes-objects/declarative-config/#how-to-create-objects

What would I do in Fleet's config file] to achieve this?

My preferred solution would be to replace this ClusterRole https://github.com/fleetdm/fleet/blob/main/tools/kubequery/kubequery-fleet.yml#L33-L44 with the ClusterRole provided in the description, and then remove all tables involving secrets. I think you'd be hard pressed to find people who were really concerned with security being happy about an external tool having blanket access to secrets.

Barring that, it's rather tough to cleanly optionally provide a toggle for this functionality because it involves both the ClusterRole as well as functionality within kubequery. You could optionally have a toggle in kubequery to enable secrets, and have instructions on how to change the ClusterRole I guess...