Unable to get lister for /v1, Resource=pods

[nipuna-perera]

What steps did you take and what happened: Updated the 0.21 from 0.19 and I am unable to see basic resources such as pods or deployments anymore. I do have access to these resources because I can use kubectl to access them.

ran octant cli and checked the browser, no resources are showing after I select namespace.

2021-06-18T15:35:50.243-0500	ERROR	api/content_manager.go:158	generate content	{"client-id": "f18ce448-d073-11eb-b482-58ef68e6c31b", "err": "generate content: unable to get Lister for /v1, Resource=pods, watcher was unable to start", "content-path": "overview/namespace/default/workloads/pods"}

The console log outputs similar messages for other resources as well and I see the following then I try to open the namespace overview. I can list namespaces however.

r
print summary tab: failed to create item view: unable to get Lister for /v1, Resource=limitranges, watcher was unable to start

What did you expect to happen: Deployments/pods and any other resources I have access to should be visible in Octant.

Anything else you would like to add: Here's some more context from the debug log

2021-06-18T15:46:21.697-0500	DEBUG	api/poller.go:84	poller is running action	{"component": "websocket-client", "client-id": "314dd248-d076-11eb-8804-58ef68e6c31b", "poller-name": "content", "poller-instance": "f2dac8ac-af6d-456e-ba1a-0e663c8577e5"}
2021-06-18T15:46:21.697-0500	DEBUG	api/poller.go:88	poller ran action	{"component": "websocket-client", "client-id": "314dd248-d076-11eb-8804-58ef68e6c31b", "poller-name": "helperManager", "poller-instance": "0a35d92e-6506-4dd3-b37d-c6e5b5ca31e0", "elapsed": "3.889µs"}
2021-06-18T15:46:21.697-0500	DEBUG	api/poller.go:88	poller ran action	{"component": "websocket-client", "client-id": "314dd248-d076-11eb-8804-58ef68e6c31b", "poller-name": "content", "poller-instance": "f2dac8ac-af6d-456e-ba1a-0e663c8577e5", "elapsed": "829ns"}
2021-06-18T15:46:22.124-0500	DEBUG	api/poller.go:84	poller is running action	{"component": "websocket-client", "client-id": "312b2d42-d076-11eb-8804-58ef68e6c31b", "poller-name": "content", "poller-instance": "d77159a2-cd89-4dce-9bc1-94c9e27a4f18"}
2021-06-18T15:46:22.125-0500	DEBUG	api/content_manager.go:185	generating content	{"client-id": "312b2d42-d076-11eb-8804-58ef68e6c31b", "contentPath": "overview/namespace/emg-pi-preprod/workloads/pods", "elapsed": "149.042µs"}
2021-06-18T15:46:22.125-0500	ERROR	api/content_manager.go:158	generate content	{"client-id": "312b2d42-d076-11eb-8804-58ef68e6c31b", "err": "generate content: unable to get Lister for /v1, Resource=pods, watcher was unable to start", "content-path": "overview/namespace/emg-pi-preprod/workloads/pods"}
github.com/vmware-tanzu/octant/internal/api.(*ContentManager).runUpdate.func1
	/private/tmp/octant-20210617-71377-4p5v5a/src/github.com/vmware-tanzu/octant/internal/api/content_manager.go:158
github.com/vmware-tanzu/octant/internal/api.(*InterruptiblePoller).Run.func1
	/private/tmp/octant-20210617-71377-4p5v5a/src/github.com/vmware-tanzu/octant/internal/api/poller.go:86
github.com/vmware-tanzu/octant/internal/api.(*InterruptiblePoller).Run
	/private/tmp/octant-20210617-71377-4p5v5a/src/github.com/vmware-tanzu/octant/internal/api/poller.go:95
github.com/vmware-tanzu/octant/internal/api.(*ContentManager).Start
	/private/tmp/octant-20210617-71377-4p5v5a/src/github.com/vmware-tanzu/octant/internal/api/content_manager.go:132
2021-06-18T15:46:22.125-0500	DEBUG	api/poller.go:88	poller ran action	{"component": "websocket-client", "client-id": "312b2d42-d076-11eb-8804-58ef68e6c31b", "poller-name": "content", "poller-instance": "d77159a2-cd89-4dce-9bc1-94c9e27a4f18", "elapsed": "342.174µs"}
2021-06-18T15:46:22.260-0500	DEBUG	api/poller.go:84	poller is running action	{"component": "websocket-client", "client-id": "312b2d42-d076-11eb-8804-58ef68e6c31b", "poller-name": "navigation", "poller-instance": "de8fff9b-7e6a-4d95-91c5-ceec27f7d8b0"}
2021-06-18T15:46:22.279-0500	DEBUG	api/poller.go:84	poller is running action	{"component": "websocket-client", "client-id": "314dd248-d076-11eb-8804-58ef68e6c31b", "poller-name": "navigation", "poller-instance": "9f1033bc-b988-4dad-8f52-70c304aec5a2"}
2021-06-18T15:46:22.281-0500	DEBUG	api/poller.go:88	poller ran action	{"component": "websocket-client", "client-id": "312b2d42-d076-11eb-8804-58ef68e6c31b", "poller-name": "navigation", "poller-instance": "de8fff9b-7e6a-4d95-91c5-ceec27f7d8b0", "elapsed": "20.498833ms"}
2021-06-18T15:46:22.296-0500	DEBUG	api/poller.go:84	poller is running action	{"component": "websocket-client", "client-id": "326297a4-d076-11eb-8804-58ef68e6c31b", "poller-name": "navigation", "poller-instance": "a12edad9-ea5c-4149-969a-36431333b5c4"}
2021-06-18T15:46:22.298-0500	DEBUG	api/poller.go:88	poller ran action	{"component": "websocket-client", "client-id": "314dd248-d076-11eb-8804-58ef68e6c31b", "poller-name": "navigation", "poller-instance": "9f1033bc-b988-4dad-8f52-70c304aec5a2", "elapsed": "19.726929ms"}
2021-06-18T15:46:22.316-0500	DEBUG	api/poller.go:88	poller ran action	{"component": "websocket-client", "client-id": "326297a4-d076-11eb-8804-58ef68e6c31b", "poller-name": "navigation", "poller-instance": "a12edad9-ea5c-4149-969a-36431333b5c4", "elapsed": "19.460078ms"}
2021-06-18T15:46:22.320-0500	DEBUG	api/poller.go:84	poller is running action	{"component": "websocket-client", "client-id": "326297a4-d076-11eb-8804-58ef68e6c31b", "poller-name": "namespaces", "poller-instance": "cfb5611a-030a-4ef6-b617-a8dea4e4e0a6"}
2021-06-18T15:46:22.365-0500	DEBUG	api/poller.go:88	poller ran action	{"component": "websocket-client", "client-id": "326297a4-d076-11eb-8804-58ef68e6c31b", "poller-name": "namespaces", "poller-instance": "cfb5611a-030a-4ef6-b617-a8dea4e4e0a6", "elapsed": "45.323432ms"}
2021-06-18T15:46:22.368-0500	DEBUG	api/poller.go:84	poller is running action	{"component": "websocket-client", "client-id": "312b2d42-d076-11eb-8804-58ef68e6c31b", "poller-name": "namespaces", "poller-instance": "230391db-366b-4600-a3a5-b46c88a0dd56"}

Environment:

• Octant version (use octant version): 0.21.0
• Kubernetes version (use kubectl version): 1.17.12 EKS
• OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc): MacOS Mojave

[wwitzel3]

We recently changed the object store for Octant to require the minimum of being able to Watch a resource.

That said, we could re-introduce the ability to query the cluster directly with List/Get and not have a cache in the middle at all, this would result in a significant slow down, but would allow you to list resources again.

[nipuna-perera]

Yes, maybe you could make that an option? In the current state, with the restrictions my company has put on resources, I am unable to see anything. It was working great for my purposes before, even though it was a bit slow.

[wwitzel3]

Ok, for this we will re-introduce the ability to start Octant using a DynamicClient directly, which will not be backed by the cache, this will allow Octant to display resources is more restricted environment that don't support Watch at the cost of speed.

For who ever takes this issue to work on it we will want a couple things:

• an option to start octant in direct client mode
• ensure that octant can be toggled to direct client mode AFTER it has started
• ensure that the poll speed is increased from every second to 5 seconds in direct client mode
• have some indicator to a client that they are in direct client mode and provide some context about what that means
• (optional) a nice experience would be to notice the watch errors for basic resources and prompt the user to try direct client mode

In the above, I think it is important to let users explicitly know they are in direct client mode via some icon indicator somewhere and provide a brief tip/explanation about what it means and the speed implications.

[nipuna-perera]

@wwitzel3 thank you! Can the poll speed be customizable in direct client mode? If someone wanted to experiment with a 3 second poll time they should be able to do so.

[ojagodzinski]

We recently changed the object store for Octant to require the minimum of being able to Watch a resource.

Is there any example of working read-only role (except secrets) for Octant? I got clusterrole with watch capability and still does not work.

[jpreese]

Wanted to ask the same. Our users only have access to a single namespace, but Octant doesn't show any resources at all. (0.21.0)

[wwitzel3]

We recently changed the object store for Octant to require the minimum of being able to Watch a resource.

Is there any example of working read-only role (except secrets) for Octant? I got clusterrole with watch capability and still does not work.

This should work if you have a clusterrole with Watch, what error are you seeing in the console output?

[wwitzel3]

Wanted to ask the same. Our users only have access to a single namespace, but Octant doesn't show any resources at all. (0.21.0)

Are you able to start Octant with the --namespace flag and provide the namespace the user has access to?

[jpreese]

Wanted to ask the same. Our users only have access to a single namespace, but Octant doesn't show any resources at all. (0.21.0)

Are you able to start Octant with the --namespace flag and provide the namespace the user has access to?

Yes. I have tried --namespace and --disable-cluster-overview

[wwitzel3]

@jpreese and @nipuna-perera I'm curious, if you run the following kubectl commands:

Check that you can watch for the namespace: kubectl auth can-i watch pods --namespace <your-namespace>

Now watch, then create a throw-away pod, or scale up a deployment, do you see the pods populate in output?

kubectl get pods --namespace <your-namespace> --watch-only

[jpreese]

Yep! All the above outputs as expected. We have admin access over the namespace so I don't think it's a permission issue there -- just lack permissions to do much outside of the namespace.

[ojagodzinski]

Same here, when service account have only access to get,list,watch on everything excepts secrets on every namespace listing anything works only on default namespace. It was working fine one version 0.16.3.

example rbac: https://github.com/kubernetes/kubernetes/issues/70387#issuecomment-434288599

KUBECONFIG=/path/kubeconfig kubectl auth can-i watch Deployment --namespace some-name
yes
KUBECONFIG=/path/kubeconfig kubectl auth can-i list Deployment --namespace some-name
yes
KUBECONFIG=/path/kubeconfig kubectl auth can-i get Deployment --namespace some-name
yes

but:

2021-11-22T12:15:54.607+0100	ERROR	describer/describer.go:95	LoadObjects	{"err": "List: CacheKey[Namespace='some-name', APIVersion='apps/v1', Kind='Deployment'] (error: unable to get Lister for /, Resource=, watcher was unable to start)"}
github.com/vmware-tanzu/octant/internal/describer.LoadObjects
	/__w/octant/octant/internal/describer/describer.go:95
github.com/vmware-tanzu/octant/internal/describer.(*ObjectLoaderFactory).LoadObjects
	/__w/octant/octant/internal/describer/describer.go:43
github.com/vmware-tanzu/octant/internal/describer.(*List).Describe
	/__w/octant/octant/internal/describer/list.go:68
github.com/vmware-tanzu/octant/internal/generator.(*Generator).Generate
	/__w/octant/octant/internal/generator/generator.go:121
github.com/vmware-tanzu/octant/internal/modules/overview.(*Overview).Content
	/__w/octant/octant/internal/modules/overview/overview.go:288
github.com/vmware-tanzu/octant/internal/api.(*ContentManager).generateContent
	/__w/octant/octant/internal/api/content_manager.go:210
github.com/vmware-tanzu/octant/internal/api.(*ContentManager).runUpdate.func1
	/__w/octant/octant/internal/api/content_manager.go:145
github.com/vmware-tanzu/octant/internal/api.(*InterruptiblePoller).Run.func1
	/__w/octant/octant/internal/api/poller.go:86
github.com/vmware-tanzu/octant/internal/api.(*InterruptiblePoller).Run
	/__w/octant/octant/internal/api/poller.go:95
github.com/vmware-tanzu/octant/internal/api.(*ContentManager).Start
	/__w/octant/octant/internal/api/content_manager.go:133