Nelo-T. Wallus
Nelo-T. Wallus
@embik Could you please assign the issue to me? Thanks!
The core issue is that the controllers that are started for workspaces register with the shared informers. The goroutines coming out of these registrations are not context-aware, so they are...
The leftover leaks in kcb-kubequota come from basically the same thing - controllers being attached to informers without deregistering when they end: https://github.com/kubernetes/kubernetes/blob/ef7d51e5ba77004cb95dc2127ed90f8bbfdabc72/pkg/controller/resourcequota/resource_quota_monitor.go#L174 Compare: https://github.com/ntnn/kubernetes/commit/8adb6cf17ff8a1f0740b69cec17e8f5caee8adae The best approach would probably...
It seems that this actually was (almost) all the leaking goroutines: 
To reproduce use the branch `kcp3350` of my kcp fork: https://github.com/ntnn/kcp/tree/kcp3350 And rewrite `k8s.io/kubernetes` to branch `kcp3350-kcp` of my kubernetes fork: https://github.com/ntnn/kubernetes/tree/kcp3350-kcp I'm gonna take a closer look at the...
Found another bug in client-go's workqueue. When a workqueue is shutdown (without draining) the `.Get` method actually keeps returning elements to drain the queue: https://github.com/ntnn/kubernetes/commit/0a59de42c9680b53dfe778fec15546b913242514 Noticed that while working on...
In addition to the fixes for upstream I'm thinking of maybe a `WorkspacedSharedInformer`, that registers its own handlers with the `SharedInformer` and maintains the handlers of the workspace controllers separately....
Exactly - and it doesn't show up because the leaked goroutines we are experiencing is from the informer which for us lives longer than those controllers, so the handlers keep...
Blocked by kcp-dev/client-go#47 After the update is done `ClusterWithContext` can be implemented to prevent the leaks.