csharp
csharp copied to clipboard
kubernetes-client
ReadNamespacedPodLog / ReadNamespacedPodLogAsync Stopped working starting in Version 5+
The following code worked fine until I updated the .NET library to 5/6:
var log = _kubernetesClient.ReadNamespacedPodLog(
podName,
currentNamespace,
container: containerName,
tailLines: returnRailLines,
timestamps: showLogTimestamps,
previous: previousLog);
The async version worked too:
var log = await _kubernetesClient.ReadNamespacedPodLogAsync(
podName,
currentNamespace,
container: containerName,
tailLines: returnRailLines,
timestamps: showLogTimestamps,
previous: previousLog);
After upgrading to 5.0.20 (and then 6.0.26), both versions of these simply block forever. Nothing is ever returned and no error is shown.
I tried upgrading to version 7.0, and then the entire app stopped working, so I couldn't test the above functionality. That may be because I'm connecting to a 1.21.9 cluster. Getting version 6.0 to work would be fine for now.
Platform: .Net 5.0 Kubectl version: Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:48:33Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"windows/amd64"} Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.9", GitCommit:"37f338aa38e0427e127162afe462e2f4150f0ba3", GitTreeState:"clean", BuildDate:"2022-02-07T20:49:26Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}
hmm I did not remember there was a change related to log and .net runtime
here is an example https://github.com/kubernetes-client/csharp/blob/master/examples/logs/Logs.cs
and test used of log https://github.com/kubernetes-client/csharp/blob/8e8619130ab5745b4a6ba0f4eaa806b8a8e544d4/tests/E2E.Tests/MnikubeTests.cs#L345
can you please confirm if the examples work well?
I tried the first example and it fails everywhere.
Brand new console app, .Net 5. I compiled it with 4.0.26, 5.0.20 & 6.0.26.
I tried them connected to an AKS cluster (see version above) and an EKS cluster:
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:48:33Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"windows/amd64"} Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.5-eks-bc4871b", GitCommit:"5236faf39f1b7a7dabea8df12726f25608131aa9", GitTreeState:"clean", BuildDate:"2021-10-29T23:32:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
They all return an internal error in the SDK:
Microsoft.Rest.HttpOperationException HResult=0x80131500 Message=Operation returned an invalid status code 'BadRequest' Source=KubernetesClient StackTrace: at k8s.Kubernetes.<ReadNamespacedPodLogWithHttpMessagesAsync>d__115.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() in //src/libraries/System.Private.CoreLib/src/System/Runtime/ExceptionServices/ExceptionDispatchInfo.cs:line 56 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) in //src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/TaskAwaiter.cs:line 173 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) in //src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/TaskAwaiter.cs:line 150 at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult() in //src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/TaskAwaiter.cs:line 551 at ConsoleApp3.Logs.<Main>d__0.MoveNext() in C:\Users\makizhne\source\repos\ConsoleApp3\ConsoleApp3\Program.cs:line 24 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() in //src/libraries/System.Private.CoreLib/src/System/Runtime/ExceptionServices/ExceptionDispatchInfo.cs:line 56 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) in //src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/TaskAwaiter.cs:line 173 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) in //src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/TaskAwaiter.cs:line 150 at System.Runtime.CompilerServices.TaskAwaiter.GetResult() in //src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/TaskAwaiter.cs:line 105 at ConsoleApp3.Logs.<Main>(String[] args)
This exception was originally thrown at this call stack:
k8s.Kubernetes.ReadNamespacedPodLogWithHttpMessagesAsync(string, string, string, bool?, bool?, int?, bool?, bool?, int?, int?, bool?, System.Collections.Generic.Dictionary<string, System.Collections.Generic.List
For reference purposes, here's an older version of my app, when this functionality worked fine:
https://scubakiz.github.io/clusterinfo/
See attached screenshots.
catch HttpOperationException ex and dump its content. it should contains what returns from server side
Microsoft.Rest.HttpOperationException: Operation returned an invalid status code 'BadRequest'
at k8s.Kubernetes.ReadNamespacedPodLogWithHttpMessagesAsync(String name, String namespaceParameter, String container, Nullable1 follow, Nullable1 insecureSkipTLSVerifyBackend, Nullable1 limitBytes, Nullable1 pretty, Nullable1 previous, Nullable1 sinceSeconds, Nullable1 tailLines, Nullable1 timestamps, Dictionary`2 customHeaders, CancellationToken cancellationToken)
at ConsoleApp3.Logs.Main(String[] args) in C:\Users\user\source\repos\ConsoleApp3\ConsoleApp3\Program.cs:line 27
print HttpOperationException.Response.Content, it should contain the error from server
the error is 401 badrequest which means client side was sending requests server could not understand
I'm betting that some query parameter got added w/ a null/invalid value when the code was regenerated, but that's just a guess.
@tg123 is right that there should be more information about why the request is considered bad in the body of the response, that will help us understand what is wrong.
I'm having the same issue as well, the method ReadNamespacedPodLogWithHttpMessagesAsync is failing with the same error code. All other functions seem to be working as expected. I believe it may be an issue related to when the pod is not created successfully, it fails to read the logs. When the pod is created successfully this call works fine. However, it is still needed in some cases to inspect the logs of a failed pod similar to the behavior of calling kubectl describe -n <namespace> <pod>.
@tg123 For context: running kubectl describe pod -n <namespace> <pod> shows me the pod, along with a list of events which include useful Warnings and Failures about the pod creation state -- that's really what I'm after, I'd like to get a useful error to surface about why the pod is not ready. The issue is that when the pod is created successfully the call to ReadNamespacedPodLogWithHttpMessagesAsync succeeds, but if there was an issue and the pod is left hung in "Pending" then this call fails and I can't inspect the logs.
To answer your question when I run kubectl logs this fails because it is an invalid command. When I run kubectl logs -n <my-namespace> -p <my-pod-name> I get the following error: Error from server (BadRequest): previous terminated container "my_container" in pod "my_pod_name" not found -- this seems similar to the exception that's being raised by the sdk.
Is there an sdk method that invokes kubectl describe pod -n <namespace> <pod> or does the ReadNamespacedPodLogWithHttpMessagesAsync call just use kubectl logs ?
@tg123 I've also tried calling ReadNamespacedEventWithHttpMessagesAsync to inspect the events of the pod after it's failed to create, and had similar results. It threw the same 'BadRequest' exception while attempting to read and watch the events from a pod.
i am porting kubectl style apis, but the process is a bit slow as i am busy with day time works
here is how describe po works, it is some data aggregation for some get apis https://github.com/kubernetes/kubectl/blob/2d67b5a3674c9c661bc03bb96cb2c0841ccee90b/pkg/describe/describe.go#L705
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue or PR as fresh with
/remove-lifecycle stale - Mark this issue or PR as rotten with
/lifecycle rotten - Close this issue or PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue or PR as fresh with
/remove-lifecycle rotten - Close this issue or PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Reopen this issue with
/reopen - Mark this issue as fresh with
/remove-lifecycle rotten - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/close not-planned
@k8s-triage-robot: Closing this issue, marking it as "Not Planned".
In response to this:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied- After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied- After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closedYou can:
- Reopen this issue with
/reopen- Mark this issue as fresh with
/remove-lifecycle rotten- Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/close not-planned
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
