"Observed a panic: "send on closed channel" while deploying pipeline on aws clusters

[rockheadjr]

Versions and Operating System

• Kubernetes version: 1.22

  1.22
  

• Tekton Pipeline version:

  Output of tkn version or kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller -o=jsonpath='{.items[0].metadata.labels.version}'

  v0.37.2 and v0.27.3 
  

• Operating System:

AML2

# Expected Behavior
run the piplines with no panic
# Actual Behavior
gets a panic
# Steps to Reproduce the Problem
ehh.. run pipeline. 

1. reinstall aws cluster (nodegroups, addons...)
2. start tkn pipeline
3. hopefully reproduce panic

# Additional Info
This issue has happened with 6 developers/6 eks clusters, at least. All at instance sizes range from t3.xl to m5.xl and enough workers for plenty of cpu and ram available. These clusters and deployments still had tekton pipeline install items correctly but output of such gets dismissed and one has to wait and use k9s to see that the pods are installed or updated. 

see panic log below. 
`E0913 13:54:17.645542     534 runtime.go:78] Observed a panic: "send on closed c
hannel" (send on closed channel)
goroutine 329 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1b40ee0, 0x1fe47b0)
      /workspace/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.
go:74 +0x95
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
      /workspace/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.
go:48 +0x89
panic(0x1b40ee0, 0x1fe47b0)
      /usr/local/go/src/runtime/panic.go:969 +0x175
github.com/tektoncd/cli/pkg/pods.(*Pod).watcher.func2(0x1d82700, 0xc000da3400, 0
x1d82700, 0xc000da3c00)
      /workspace/src/$(params.package)/pkg/pods/pod.go:107 +0x48
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate(...)
      /workspace/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:23
4
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
      /workspace/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.
go:775 +0x1c5
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00082bf60)
      /workspace/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
+0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00096ff60, 0x2006120, 0xc00082
d0e0, 0x1b01501, 0xc00075b3e0)
      /workspace/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
+0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00082bf60, 0x3b9aca00, 0x0, 0x1
, 0xc00075b3e0)
      /workspace/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
+0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
      /workspace/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0007bbf80)
      /workspace/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.
go:771 +0x95
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000576450, 0xc0002d336
0)
      /workspace/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73
+0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
      /workspace/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:71
+0x65
panic: send on closed channel [recovered]
      panic: send on closed channel
goroutine 329 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
      /workspace/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.
go:55 +0x10c
panic(0x1b40ee0, 0x1fe47b0)
      /usr/local/go/src/runtime/panic.go:969 +0x175
github.com/tektoncd/cli/pkg/pods.(*Pod).watcher.func2(0x1d82700, 0xc000da3400, 0
x1d82700, 0xc000da3c00)
      /workspace/src/$(params.package)/pkg/pods/pod.go:107 +0x48
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate(...)
      /workspace/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:23
4
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
      /workspace/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.
go:775 +0x1c5
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00082bf60)
      /workspace/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
+0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00055ff60, 0x2006120, 0xc00082
d0e0, 0x1b01501, 0xc00075b3e0)
      /workspace/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00082bf60, 0x3b9aca00, 0x0, 0x1, 0xc00075b3e0)
      /workspace/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
      /workspace/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0007bbf80)
      /workspace/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:771 +0x95
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000576450, 0xc0002d3360)
      /workspace/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
      /workspace/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:71 +0x65
Bootstrap pipeline bootstrap-pipeline deployment failed`

[piyush-garg]

@rockheadjr which tkn command is causing this?

[sudo-bmitch]

I think this is the same panic as #1650.

[rockheadjr]

@piyush-garg i will check today

[rockheadjr]

@piyush-garg tkn pipeline start

[piyush-garg]

@rockheadjr if it is similar to #1650 then we already have the fix in review

[tekton-robot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale with a justification. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close with a justification. If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten with a justification. Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with /close with a justification. If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen with a justification. Mark the issue as fresh with /remove-lifecycle rotten with a justification. If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen with a justification. Mark the issue as fresh with /remove-lifecycle rotten with a justification. If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.