Lens metrics fails to install (node-exporter) on GKE autopilot cluster

[jim-docker]

[0] error:   ▪ [RESOURCE-APPLIER] kubectl errored: Command failed: /Users/*****/Library/Application Support/LensDev/binaries/kubectl/1.24.8/kubectl apply --kubeconfig /var/folders/3d/vq292d5n71gbkjs0cnwzw3280000gq/T/kubeconfig-38d0c8a1ce457455addfb5e7f6cb0045 --prune -l app.kubernetes.io/name=lens-metrics -f /private/var/folders/3d/vq292d5n71gbkjs0cnwzw3280000gq/T/9798f85155cf0187f2803dc40f4bba29
[0] Warning: Autopilot increased resource requests for Deployment lens-metrics/kube-state-metrics to meet requirements. See http://g.co/gke/autopilot-resources
[0] Warning: Autopilot set default resource requests on StatefulSet lens-metrics/prometheus for container chown, as resource requests were not specified, and adjusted resource requests to meet requirements. See http://g.co/gke/autopilot-defaults and http://g.co/gke/autopilot-resources
[0] Warning: Autopilot increased resource requests for DaemonSet lens-metrics/node-exporter to meet requirements. See http://g.co/gke/autopilot-resources
[0] Error from server (Forbidden): error when creating "/private/var/folders/3d/vq292d5n71gbkjs0cnwzw3280000gq/T/9798f85155cf0187f2803dc40f4bba29/10.yaml": clusterroles.rbac.authorization.k8s.io is forbidden: User "*****" cannot create resource "clusterroles" in API group "rbac.authorization.k8s.io" at the cluster scope: requires one of ["container.clusterRoles.create"] permission(s).
[0] Error from server (Forbidden): error when creating "/private/var/folders/3d/vq292d5n71gbkjs0cnwzw3280000gq/T/9798f85155cf0187f2803dc40f4bba29/12.yaml": clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "*****" cannot create resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope: requires one of ["container.clusterRoleBindings.create"] permission(s).
[0] Error from server (Forbidden): error when creating "/private/var/folders/3d/vq292d5n71gbkjs0cnwzw3280000gq/T/9798f85155cf0187f2803dc40f4bba29/6.yaml": clusterroles.rbac.authorization.k8s.io is forbidden: User "*****" cannot create resource "clusterroles" in API group "rbac.authorization.k8s.io" at the cluster scope: requires one of ["container.clusterRoles.create"] permission(s).
[0] Error from server (Forbidden): error when creating "/private/var/folders/3d/vq292d5n71gbkjs0cnwzw3280000gq/T/9798f85155cf0187f2803dc40f4bba29/7.yaml": clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "*****" cannot create resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope: requires one of ["container.clusterRoleBindings.create"] permission(s).
[0] Error from server (GKE Warden constraints violations): error when creating "/private/var/folders/3d/vq292d5n71gbkjs0cnwzw3280000gq/T/9798f85155cf0187f2803dc40f4bba29/8.yaml": admission webhook "gkepolicy.common-webhooks.networking.gke.io" denied the request: GKE Warden rejected the request because it violates one or more constraints.
[0] Violations details: {"[denied by autogke-disallow-hostnamespaces]":["enabling hostPID is not allowed in Autopilot."],"[denied by autogke-no-write-mode-hostpath]":["hostPath volume proc used in container node-exporter uses path /proc which is not allowed in Autopilot. Allowed path prefixes for hostPath volumes are: [/var/log/].","hostPath volume sys used in container node-exporter uses path /sys which is not allowed in Autopilot. Allowed path prefixes for hostPath volumes are: [/var/log/].","hostPath volume root used in container node-exporter uses path / which is not allowed in Autopilot. Allowed path prefixes for hostPath volumes are: [/var/log/]."]}

few other problems maybe too

[jim-docker]

The issue with resource creation (clusterroles, clusterrolebindings) can be resolved by running:

kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=<user-name>

where <user-name> is the user associated with the kube context you are using (look at the kubecong, kubectl config view)

If creating the clusterrolebinding fails, i.e.

User "<user-name>" cannot create clusterrolebindings.rbac.authorization.k8s.io at the cluster scope.: 
  "Required \"container.clusterRoleBindings.create\" permission." 
  (post clusterrolebindings.rbac.authorization.k8s.io)

then you first need more permissions at the GCP level, in the associated project, for you as user [email protected]. You, or someone with higher permissions, must run

gcloud projects add-iam-policy-binding $PROJECT \
  --member=user:[email protected] \
  --role=roles/container.admin

Then retry creating the clusterrolebinding

See https://stackoverflow.com/a/49784455

[jim-docker]

The remaining issue is related to node-exporter trying to access / and /proc and /sys which is not allowed by autopilot. The ultimate solution may be to instead support one of the GCP managed services for metrics (See https://stackoverflow.com/a/73066153)

[celvin]

did anyone make it work? It installs but it still doesn't show any Metrics