kubectl
kubectl copied to clipboard
kubernetes
`kubectl top` command returns "Error from server (NotFound): pods ..." when pod is in init stage
What happened?
I've executed kubectl top pod <pod-name> while the pod is initizalizing state. Instead of returning the resource consumption the server replied with:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
scanner-db-7484ff5c96-dsgg9 0/1 Init:0/1 0 3m37s
❯ kubectl top pod scanner-db-7484ff5c96-dsgg9
Error from server (NotFound): pods "scanner-db-7484ff5c96-dsgg9" not found
On the other hand kubectl get logs scanner-db-7484ff5c96-dsgg9 init-db does work and print the logs correclty.
What did you expect to happen?
I've expected to the pod's memory and CPU consumption from the top command.
How can we reproduce it (as minimally and precisely as possible)?
- Create a Pod with a long running init-container (i.e. sleep 10000)
- Run
kubectl top pod <pod-name>on that Pod - Pod not found error is returned.
Anything else we need to know?
No response
Kubernetes version
$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.1", GitCommit:"17b7accf8fd25125ce015cf4bea7d3cd3f336317", GitTreeState:"clean", BuildDate:"2023-08-11T20:28:13Z", GoVersion:"go1.19.10 X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.14+20cda61", GitCommit:"3bdfba0be09da2bfdef3b63e421e6a023bbb08e6", GitTreeState:"clean", BuildDate:"2023-10-13T19:46:17Z", GoVersion:"go1.19.10 X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}
Cloud provider
OS version
# On Linux:
$ cat /etc/os-release
NAME="Fedora Linux"
VERSION="38 (Workstation Edition)"
ID=fedora
VERSION_ID=38
VERSION_CODENAME=""
PLATFORM_ID="platform:f38"
PRETTY_NAME="Fedora Linux 38 (Workstation Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:38"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f38/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=38
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=38
SUPPORT_END=2024-05-14
VARIANT="Workstation Edition"
VARIANT_ID=workstation
$ uname -a
Linux <name> 6.4.15-200.fc38.x86_64 kubernetes/kubernetes#1 SMP PREEMPT_DYNAMIC Thu Sep 7 00:25:01 UTC 2023 x86_64 GNU/Linux
Install tools
Container runtime (CRI) and version (if applicable)
Related plugins (CNI, CSI, ...) and versions (if applicable)
This issue is currently awaiting triage.
If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.
The triage/accepted label can be added by org members by writing /triage accepted in a comment.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
On the other hand kubectl get logs scanner-db-7484ff5c96-dsgg9 init-db does work and print the logs correclty.
I suppose you mean kubectl logs scanner-db-7484ff5c96-dsgg9 init-db
can you send the log here?
(if this is actually a bug in something else, eg metrics-server, then let's track it in the right place and not in k/kubectl)
I think that kubectl top could and maybe should wait for metrics to appear for the Pod, by default. As soon as a Pod exists then kubectl top should keep running even if metrics are not yet available.
Related but separate, I'd expect to see metrics for Pods that are still starting up (ie: that either an init container is running, or a sidecar container is starting).
Do you mean you can't replicate the issue? the issue is already open @Ithrael
Do you mean you can't replicate the issue? the issue is already open @Ithrael
I'm sorry, there was an error with my previous test case. I just tried again, and I can replicate the issue.
kubectl top po nginx-5678f599b4-77rjp -v=9
log:
GET https://127.0.0.1:63587/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/nginx-5678f599b4-77rjp 404 Not Found in 3 milliseconds
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue as fresh with
/remove-lifecycle stale - Close this issue with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
