Cluster stuck on "updating"

[samirmajen]

Describe the bug Every page on the pixie UI shows Cluster '' is updating and is therefore unusable.

To Reproduce Steps to reproduce the behavior: Install self hosted pixie on a kubernetes cluster

Expected behavior The UI should work

Logs

Artifact Tracker Server:

time="2023-07-26T10:59:51Z" level=warning msg="finished unary call with code Internal" error="rpc error: code = Internal desc = failed to get URL" fields.time=383.871059ms grpc.code=Internal grpc.method=GetDownloadLink grpc.service=px.services.ArtifactTracker grpc.start_time="2023-07-26T10:59:50Z" peer.address="100.64.1.122:54588" span.kind=server system=grpc

Config Manager Server:

time="2023-07-26T10:59:50Z" level=info msg="Fetching config for Vizier" time="2023-07-26T10:59:51Z" level=error msg="Failed to fetch Vizier templates" error="rpc error: code = Internal desc = failed to get URL" time="2023-07-26T10:59:51Z" level=warning msg="finished unary call with code Internal" error="rpc error: code = Internal desc = failed to get URL" fields.time=386.06767ms grpc.code=Internal grpc.method=GetConfigForVizier grpc.service=px.services.ConfigManagerService grpc.start_time="2023-07-26T10:59:50Z" peer.address="100.64.9.181:60984" span.kind=server system=grpc Logs from 7/26/2023

App information (please complete the following information):

• Pixie version: 0.1.6
• K8s cluster version: EKS 1.27.1
• Node Kernel version: 5.10.179-168.710.amzn2.x86_64
• Browser version: Chrome 108

Additional context Add any other context about the problem here.

[samirmajen]

looks like this change https://github.com/pixie-io/pixie/pull/1601 fixes the artifact tracker issue, cluster is still showing the "updating" state and the vizier operator now coredumps:

goroutine 1 [running]: k8s.io/client-go/discovery.convertAPIResource(...) external/io_k8s_client_go/discovery/aggregated_discovery.go:88 k8s.io/client-go/discovery.convertAPIGroup({{{0x0, 0x0}, {0x0, 0x0}}, {{0xc0001674e8, 0x15}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...}) external/io_k8s_client_go/discovery/aggregated_discovery.go:69 +0x570 k8s.io/client-go/discovery.SplitGroupsAndResources({{{0xc00056c000, 0x15}, {0xc0006003a0, 0x1b}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...}) external/io_k8s_client_go/discovery/aggregated_discovery.go:35 +0x118 k8s.io/client-go/discovery.(*DiscoveryClient).downloadAPIs(0x34f9434?) external/io_k8s_client_go/discovery/discovery_client.go:310 +0x47c k8s.io/client-go/discovery.(*DiscoveryClient).GroupsAndMaybeResources(0x362d993?) external/io_k8s_client_go/discovery/discovery_client.go:198 +0x5c k8s.io/client-go/discovery.ServerGroupsAndResources({0x15e31f8, 0xc0001f39e0}) external/io_k8s_client_go/discovery/discovery_client.go:392 +0x59 k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources.func1() external/io_k8s_client_go/discovery/discovery_client.go:356 +0x25 k8s.io/client-go/discovery.withRetries(0x2, 0xc0008670b0) external/io_k8s_client_go/discovery/discovery_client.go:621 +0x71 k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources(0x0?) external/io_k8s_client_go/discovery/discovery_client.go:355 +0x3a k8s.io/client-go/restmapper.GetAPIGroupResources({0x15e31f8?, 0xc0001f39e0?}) external/io_k8s_client_go/restmapper/discovery.go:148 +0x42 sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper.func1() external/io_k8s_sigs_controller_runtime/pkg/client/apiutil/dynamicrestmapper.go:94 +0x25 sigs.k8s.io/controller-runtime/pkg/client/apiutil.(*dynamicRESTMapper).setStaticMapper(...) external/io_k8s_sigs_controller_runtime/pkg/client/apiutil/dynamicrestmapper.go:130 sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper(0xc000568510?, {0x0, 0x0, 0x568e81b69d343b01?}) external/io_k8s_sigs_controller_runtime/pkg/client/apiutil/dynamicrestmapper.go:110 +0x182 sigs.k8s.io/controller-runtime/pkg/cluster.setOptionsDefaults.func1(0xc0004fbe30?) external/io_k8s_sigs_controller_runtime/pkg/cluster/cluster.go:217 +0x25 sigs.k8s.io/controller-runtime/pkg/cluster.New(0xc00045bd40, {0xc00070fa58, 0x1, 0x0?}) external/io_k8s_sigs_controller_runtime/pkg/cluster/cluster.go:159 +0x18d sigs.k8s.io/controller-runtime/pkg/manager.New(_, {0xc0004fbe30, 0x0, 0x0, {{0x15dd650, 0xc000522e40}, 0x0}, 0x0, {0x0, 0x0}, ...}) external/io_k8s_sigs_controller_runtime/pkg/manager/manager.go:351 +0xf9 main.main() src/operator/manager.go:62 +0x219