Failed to run exporter on OpenShift 4.x

[hrytsai]

Describe the bug When run kor in cluster with exporter flag kor -exporter --resources secret,configmap,pvc -n kor-test-namespace We got an error: Failed to retrieve namespaces: namespaces is forbidden: User "system:serviceaccount:kor-test-namespace:dev-kor" cannot list resource "namespaces" in API group "" at the cluster scope We faced similiar issue when run kor v0.3.5 without exporter flag, see https://github.com/yonahd/kor/issues/213

To Reproduce Run kor in cluster with options: -exporter --resources secret,configmap,pvc -n <namespace_name>

Expected behavior Namespace will be scanned and result provided without get access to list of namespaces, get only unused resources in provided namespace

OS version, architecture and kor version OpenShift 4.12 Kor version: 0.3.8

Additional context For example: kubectl get ns will return

Error from server (Forbidden): namespaces is forbidden: User "kor-test-user" cannot list resource "namespaces" in API group "" at the cluster scope

but when you run: kubectl get project you will get result:

NAME DISPLAY NAME STATUS ... kor-test-namespace ... This issue also might be connected to: https://github.com/yonahd/kor/issues/218

[yonahd]

Strange that this came back Looking what changed

[yonahd]

@hrytsai this worked fine in 0.3.7?

[hrytsai]

@hrytsai this worked fine in 0.3.7?

Nope, the same problem

[yonahd]

Very strange I cannot seem to replicate this error. There are 2 places in the code that call the list namespaces api and your command hits neither of them. Let me see if anyone else can replicate

[doronkg]

The issue results from the retrieveUsedClusterRoles() function that is called in pkg/kor/clusrterroles.go.

func retrieveUsedClusterRoles(clientset kubernetes.Interface, filterOpts *filters.Options) ([]string, error) {

	//Get a list of all namespaces
	namespaceList, err := clientset.CoreV1().Namespaces().List(context.TODO(), metav1.ListOptions{})
	if err != nil {
		fmt.Fprintf(os.Stderr, "Failed to retrieve namespaces: %v\n", err)
		os.Exit(1)
	}
	roleBindingsAllNameSpaces := make([]v1.RoleBinding, 0)
	
	...

When either command kor all or kor exporter is used, GetUnusedAll() function is invoked under pkg/kor/all.go. In kor versions up to v0.3.8 including, there was an issue with this function, it also scanned the cluster for non-namespaced resources (i.e. CRD, PV, ClusterRole). It was reported in #178.

As getting unused ClusterRoles requires access to all namespaces, the following error raised:

Failed to retrieve namespaces: namespaces is forbidden: User "system:serviceaccount:kor-test-namespace:dev-kor" cannot list resource "namespaces" in API group "" at the cluster scope

This issue was fixed in #258, with the following function correction:

func GetUnusedAll(filterOpts *filters.Options, clientset kubernetes.Interface, apiExtClient apiextensionsclientset.Interface, dynamicClient dynamic.Interface, outputFormat string, opts Opts) (string, error) {
	unusedAllNamespaced, err := GetUnusedAllNamespaced(filterOpts, clientset, outputFormat, opts)
	if err != nil {
		fmt.Printf("err: %v\n", err)
	}

	// Skip getting non-namespaced resources if --include-namespaces flag is used
	if len(filterOpts.IncludeNamespaces) > 0 {
		return unusedAllNamespaced, nil
	}

	...

@yonahd as this fix was merged to main, let's bump v0.3.9, it should resolve the issue mentioned in this case.

[yonahd]

The issue results from the retrieveUsedClusterRoles() function that is called in pkg/kor/clusrterroles.go.

func retrieveUsedClusterRoles(clientset kubernetes.Interface, filterOpts *filters.Options) ([]string, error) {

	//Get a list of all namespaces
	namespaceList, err := clientset.CoreV1().Namespaces().List(context.TODO(), metav1.ListOptions{})
	if err != nil {
		fmt.Fprintf(os.Stderr, "Failed to retrieve namespaces: %v\n", err)
		os.Exit(1)
	}
	roleBindingsAllNameSpaces := make([]v1.RoleBinding, 0)
	
	...

When either command kor all or kor exporter is used, GetUnusedAll() function is invoked under pkg/kor/all.go. In kor versions up to v0.3.8 including, there was an issue with this function, it also scanned the cluster for non-namespaced resources (i.e. CRD, PV, ClusterRole). It was reported in #178.

As getting unused ClusterRoles requires access to all namespaces, the following error raised:

Failed to retrieve namespaces: namespaces is forbidden: User "system:serviceaccount:kor-test-namespace:dev-kor" cannot list resource "namespaces" in API group "" at the cluster scope

This issue was fixed in #258, with the following function correction:

func GetUnusedAll(filterOpts *filters.Options, clientset kubernetes.Interface, apiExtClient apiextensionsclientset.Interface, dynamicClient dynamic.Interface, outputFormat string, opts Opts) (string, error) {
	unusedAllNamespaced, err := GetUnusedAllNamespaced(filterOpts, clientset, outputFormat, opts)
	if err != nil {
		fmt.Printf("err: %v\n", err)
	}

	// Skip getting non-namespaced resources if --include-namespaces flag is used
	if len(filterOpts.IncludeNamespaces) > 0 {
		return unusedAllNamespaced, nil
	}

	...

@yonahd as this fix was merged to main, let's bump v0.3.9, it should resolve the issue mentioned in this case.

@doronkg the all function should not be called in the above case. This is using the --resources flag which triggers multi https://github.com/yonahd/kor/blob/v0.3.8/pkg/kor/exporter.go#L86

[doronkg]

@hrytsai this worked fine in 0.3.7?

Nope, the same problem

Let's split it into two cases:

1) The --resources flag was only introduced in v0.3.8, so it couldn't be reproduced in v0.3.7 with that flag. If it failed for you in v0.3.7, then that's the case mentioned in this comment and without the flag (or it would result with unkown flag error), that's been worked around in v0.3.8 (only if --resources flag is used). I've been able to reproduce it.

Without the flag, not only the error message will still be displayed, the pod will enter a CrashLoopBackOff state.

Example A - All

$ oc get pods -n kor
NAME                            READY   STATUS             RESTARTS      AGE
kor-exporter-85d5d69cc6-vwbz5   0/1     CrashLoopBackOff   5 (64s ago)   4m30s

$ oc logs kor-exporter-85d5d69cc6-vwbz5 -n kor
Server listening on :8080
collecting unused resources
Failed to retrieve namespaces: namespaces is forbidden: User "system:serviceaccount:kor:test" cannot list resource "namespaces" in API group "" at the cluster scope

$ oc get pod kor-exporter-85d5d69cc6-vwbz5 -n kor -o yaml
apiVersion: v1
kind: Pod
metadata:
  name: kor-exporter-85d5d69cc6-vwbz5
  namespace: kor
spec:
  containers:
  - args:
    - exporter
    command:
    - kor
    image: yonahdissen/kor:latest
    imagePullPolicy: Always
    name: kor-exporter-container
...

Example B - Resources by Namespace

$ oc get pods -n kor
NAME                            READY   STATUS             RESTARTS      AGE
kor-exporter-6b9fdc9f4f-svdgb   1/1     Running   0          28s

$ oc logs kor-6b9fdc9f4f-svdgb -n kor
Server listening on :8080
collecting unused resources

$ oc get pod kor-exporter-6b9fdc9f4f-svdgb -n kor -o yaml
apiVersion: v1
kind: Pod
metadata:
  name: kor-exporter-6b9fdc9f4f-svdgb
  namespace: kor
 spec:
    containers:
    - args:
      - exporter
      - --resources
      - pvc,secret
      - -n
      - kor
      command:
      - kor
      image: yonahdissen/kor:latest
      imagePullPolicy: Always
      name: kor-exporter-container
...

2) In v0.3.8, as said, this error message shouldn't be displayed (see comment) if the flag is used, but if the used ServiceAccount doesn't have get permissions to the requested namespaces it should query, this would be displayed in the container:

NOTE: That depends whether the ClusterRoleBinding from the Helm chart was applied (allows to get/list/watch all namespaces by default) or rather the used ServiceAccount is binded with custom RBAC.

$ oc logs kor-exporter-56d6696cf7-zqdn6 -n kor
Server listening on :8080
collecting unused resources
namespace [kor-test-namespace] not found

That behavior results from Namespaces() function in options.go that handles the --include-namespaces / -n flag.

func (o *Options) Namespaces(clientset kubernetes.Interface) []string {
	...
	
	for _, ns := range includeNamespaces {

		_, err := clientset.CoreV1().Namespaces().Get(context.TODO(), ns, metav1.GetOptions{})
		if err == nil {
			namespacesMap[ns] = true
		} else {
			fmt.Fprintf(os.Stderr, "namespace [%s] not found\n", ns)
		}
	}

---

@hrytsai if you encounter case (1), make sure to upgrade to v0.3.8 and have the pod restarted. If you've upgraded and the error message is still displayed, let us know here. If you've upgraded and case (2) is displayed, follow these steps:

$ oc auth can-i get ns/kor-test-namespace --as=system:serviceaccount:kor-test-namespace:dev-kor
Warning: resource 'namespaces' is not namespace scoped
no

This could be easily resolved by manually granting the following permissions to the ServiceAccount (if not using the Helm chart), with designated ClusterRole & ClusterRoleBinding:

rules:
  - apiGroups: [""]
    resources:
      - namespaces
    verbs:
      - get
      - list
      - watch    

Once you've made the change, run the following command that should display:

$ oc auth can-i get ns/kor-test-namespace --as=system:kor-test-namespace:dev-kor
Warning: resource 'namespaces' is not namespace scoped
yes

Restart the pod, and validate that the warning no longer displays.

Please let me know how it went.

[hrytsai]

Hi Thanks for explanation, we have case 2. But problem is that we cannot provide SA\User for get, list, watch namespaces permission in multi tenancy cluster for security requirements. We operate concept "Project" in OpenShift that is tightly connected to "Namespace" and SA\Users can get necessary permission. Is it possible to add option like: isOpenShift and use "Project" instead of "Namespaces" in this case in application ?

[doronkg]

Hi

Thanks for explanation, we have case 2.

But problem is that we cannot provide SA\User for get, list, watch namespaces permission in multi tenancy cluster for security requirements.

We operate concept "Project" in OpenShift that is tightly connected to "Namespace" and SA\Users can get necessary permission.

Is it possible to add option like:

isOpenShift and use "Project" instead of "Namespaces" in this case in application ?

I'm glad we've got to the bottom of this. We'll look into an OpenShift Project-compatible solution and update you.

If you have additional feedbacks regarding kor on OpenShift, we'd be happy to hear.