kubernetes-reflector icon indicating copy to clipboard operation
kubernetes-reflector copied to clipboard

Published 20 hours ago • verified publisher icon emberstack

Reflector hangs after exception

Open loganmzz opened this issue 2 months ago • 11 comments

Versions:

  • Kubernetes: v1.28.9-eks-036c24b
  • Reflector: 7.0.193

We encountered issue with outdated certificate for some services. After investigation, it seems Reflector is hanging since several weeks, not mirroring update from Cert Manager certificate secret.

Tried to update an annotation on given secret, but no reaction. However, HTTP healthcheck is still responding.

Last logs:

2024-04-22 14:58:36.937 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretWatcher) Requesting V1Secret resources
2024-04-22 14:58:37.305 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Auto-reflected argocd/argo-workflows-s3 where permitted. Created 0 - Updated 0 - Deleted 0 - Validated 51.
2024-04-22 14:58:37.594 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Auto-reflected cert-manager/wildcard-priv-mycompany-cloud where permitted. Created 0 - Updated 0 - Deleted 0 - Validated 82.
2024-04-22 14:58:37.803 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Auto-reflected global/global-secret where permitted. Created 0 - Updated 0 - Deleted 0 - Validated 51.
2024-04-22 15:13:45.131 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretWatcher) Session closed. Duration: 00:15:08.1942289. Faulted: False.
2024-04-22 15:13:45.132 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretWatcher) Requesting V1Secret resources
2024-04-22 15:13:45.145 +00:00 [INF] (ES.Kubernetes.Reflector.Core.NamespaceWatcher) Session closed. Duration: 00:19:31.9075751. Faulted: False.
2024-04-22 15:13:45.146 +00:00 [INF] (ES.Kubernetes.Reflector.Core.NamespaceWatcher) Requesting V1Namespace resources
2024-04-22 15:13:45.668 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Auto-reflected argocd/argo-workflows-s3 where permitted. Created 0 - Updated 0 - Deleted 0 - Validated 51.
2024-04-22 15:13:45.902 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Auto-reflected cert-manager/wildcard-priv-mycompany-cloud where permitted. Created 0 - Updated 0 - Deleted 0 - Validated 82.
2024-04-22 15:13:46.104 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Auto-reflected global/global-secret where permitted. Created 0 - Updated 0 - Deleted 0 - Validated 51.
2024-04-22 15:18:15.767 +00:00 [ERR] (ES.Kubernetes.Reflector.Core.NamespaceWatcher) Faulted due to exception.
System.IO.IOException: The request was aborted.
 ---> System.IO.IOException: The response ended prematurely while waiting for the next frame from the server.
   at System.Net.Http.Http2Connection.<ReadFrameAsync>g__ThrowMissingFrame|69_1()
   at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame)
   at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
   at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
   at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span`1 buffer, Boolean partOfSyncRead)
   at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)
   at k8s.LineSeparatedHttpContent.CancelableStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at System.IO.StreamReader.ReadBufferAsync(CancellationToken cancellationToken)
   at System.IO.StreamReader.ReadLineAsyncInternal(CancellationToken cancellationToken)
   at k8s.Watcher`1.<>c.<CreateWatchEventEnumerator>b__21_1[TR](Task`1 t)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at k8s.Watcher`1.CreateWatchEventEnumerator(Func`1 streamReaderCreator, Action`1 onError, CancellationToken cancellationToken)+MoveNext()
   at k8s.Watcher`1.CreateWatchEventEnumerator(Func`1 streamReaderCreator, Action`1 onError, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at ES.Kubernetes.Reflector.Core.Watchers.WatcherBackgroundService`2.ExecuteAsync(CancellationToken stoppingToken) in /src/ES.Kubernetes.Reflector/Core/Watchers/WatcherBackgroundService.cs:line 78
   at ES.Kubernetes.Reflector.Core.Watchers.WatcherBackgroundService`2.ExecuteAsync(CancellationToken stoppingToken) in /src/ES.Kubernetes.Reflector/Core/Watchers/WatcherBackgroundService.cs:line 78
2024-04-22 15:18:15.772 +00:00 [INF] (ES.Kubernetes.Reflector.Core.NamespaceWatcher) Session closed. Duration: 00:04:30.6266915. Faulted: True.
2024-04-22 15:18:15.770 +00:00 [ERR] (ES.Kubernetes.Reflector.Core.ConfigMapWatcher) Faulted due to exception.
System.IO.IOException: The request was aborted.
 ---> System.IO.IOException: The response ended prematurely while waiting for the next frame from the server.
   at System.Net.Http.Http2Connection.<ReadFrameAsync>g__ThrowMissingFrame|69_1()
   at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame)
   at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
   at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
   at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span`1 buffer, Boolean partOfSyncRead)
   at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)
   at k8s.LineSeparatedHttpContent.CancelableStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at System.IO.StreamReader.ReadBufferAsync(CancellationToken cancellationToken)
   at System.IO.StreamReader.ReadLineAsyncInternal(CancellationToken cancellationToken)
   at k8s.Watcher`1.<>c.<CreateWatchEventEnumerator>b__21_1[TR](Task`1 t)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at k8s.Watcher`1.CreateWatchEventEnumerator(Func`1 streamReaderCreator, Action`1 onError, CancellationToken cancellationToken)+MoveNext()
   at k8s.Watcher`1.CreateWatchEventEnumerator(Func`1 streamReaderCreator, Action`1 onError, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at ES.Kubernetes.Reflector.Core.Watchers.WatcherBackgroundService`2.ExecuteAsync(CancellationToken stoppingToken) in /src/ES.Kubernetes.Reflector/Core/Watchers/WatcherBackgroundService.cs:line 78
   at ES.Kubernetes.Reflector.Core.Watchers.WatcherBackgroundService`2.ExecuteAsync(CancellationToken stoppingToken) in /src/ES.Kubernetes.Reflector/Core/Watchers/WatcherBackgroundService.cs:line 78
2024-04-22 15:18:15.774 +00:00 [INF] (ES.Kubernetes.Reflector.Core.ConfigMapWatcher) Session closed. Duration: 00:23:19.9873238. Faulted: True.
2024-04-22 15:18:15.769 +00:00 [ERR] (ES.Kubernetes.Reflector.Core.SecretWatcher) Faulted due to exception.
System.IO.IOException: The request was aborted.
 ---> System.IO.IOException: The response ended prematurely while waiting for the next frame from the server.
   at System.Net.Http.Http2Connection.<ReadFrameAsync>g__ThrowMissingFrame|69_1()
   at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame)
   at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
   at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
   at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span`1 buffer, Boolean partOfSyncRead)
   at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)
   at k8s.LineSeparatedHttpContent.CancelableStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at System.IO.StreamReader.ReadBufferAsync(CancellationToken cancellationToken)
   at System.IO.StreamReader.ReadLineAsyncInternal(CancellationToken cancellationToken)
   at k8s.Watcher`1.<>c.<CreateWatchEventEnumerator>b__21_1[TR](Task`1 t)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at k8s.Watcher`1.CreateWatchEventEnumerator(Func`1 streamReaderCreator, Action`1 onError, CancellationToken cancellationToken)+MoveNext()
   at k8s.Watcher`1.CreateWatchEventEnumerator(Func`1 streamReaderCreator, Action`1 onError, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at ES.Kubernetes.Reflector.Core.Watchers.WatcherBackgroundService`2.ExecuteAsync(CancellationToken stoppingToken) in /src/ES.Kubernetes.Reflector/Core/Watchers/WatcherBackgroundService.cs:line 78
   at ES.Kubernetes.Reflector.Core.Watchers.WatcherBackgroundService`2.ExecuteAsync(CancellationToken stoppingToken) in /src/ES.Kubernetes.Reflector/Core/Watchers/WatcherBackgroundService.cs:line 78
2024-04-22 15:18:15.781 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretWatcher) Session closed. Duration: 00:04:30.6497023. Faulted: True.
2024-04-22 15:18:20.772 +00:00 [INF] (ES.Kubernetes.Reflector.Core.ConfigMapWatcher) Requesting V1ConfigMap resources
2024-04-22 15:18:20.772 +00:00 [INF] (ES.Kubernetes.Reflector.Core.NamespaceWatcher) Requesting V1Namespace resources
2024-04-22 15:18:20.783 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretWatcher) Requesting V1Secret resources
2024-04-22 15:18:21.484 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Auto-reflected argocd/argo-workflows-s3 where permitted. Created 0 - Updated 0 - Deleted 0 - Validated 51.
2024-04-22 15:18:21.873 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Auto-reflected cert-manager/wildcard-priv-mycompany-cloud where permitted. Created 0 - Updated 0 - Deleted 0 - Validated 82.
2024-04-22 15:18:22.057 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Auto-reflected global/global-secret where permitted. Created 0 - Updated 0 - Deleted 0 - Validated 51.

loganmzz avatar May 24 '24 14:05 loganmzz