cockroach-operator icon indicating copy to clipboard operation
cockroach-operator copied to clipboard

Published 20 hours ago • verified publisher icon cockroachdb

Logging panic during e2e

Open chrislovecnm opened this issue 4 years ago • 0 comments

I am seeing:

2021-05-27T18:51:12.9562560Z E0527 18:51:12.875270      23 runtime.go:76] Observed a panic: Log in goroutine after TestUpgradesMajorVersion20_1To20_2 has completed
2021-05-27T18:51:12.9564037Z goroutine 3016 [running]:
2021-05-27T18:51:12.9564874Z k8s.io/apimachinery/pkg/util/runtime.logPanic(0x19d1920, 0xc00095d670)
2021-05-27T18:51:12.9566123Z 	external/io_k8s_apimachinery/pkg/util/runtime/runtime.go:74 +0x95
2021-05-27T18:51:12.9567460Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
2021-05-27T18:51:12.9568612Z 	external/io_k8s_apimachinery/pkg/util/runtime/runtime.go:48 +0x86
2021-05-27T18:51:12.9569939Z panic(0x19d1920, 0xc00095d670)
2021-05-27T18:51:12.9570511Z 	GOROOT/src/runtime/panic.go:965 +0x1b9
2021-05-27T18:51:12.9571405Z testing.(*common).logDepth(0xc00252af00, 0xc003b4c9a0, 0xa6, 0x3)
2021-05-27T18:51:12.9572058Z 	GOROOT/src/testing/testing.go:765 +0x5df
2021-05-27T18:51:12.9572567Z testing.(*common).log(...)
2021-05-27T18:51:12.9573076Z 	GOROOT/src/testing/testing.go:747
2021-05-27T18:51:12.9573697Z testing.(*common).Logf(0xc00252af00, 0x1c5639e, 0x2, 0xc00095d660, 0x1, 0x1)
2021-05-27T18:51:12.9574335Z 	GOROOT/src/testing/testing.go:793 +0x7e
2021-05-27T18:51:12.9575314Z go.uber.org/zap/zaptest.testingWriter.Write(0x1ef7598, 0xc00252af00, 0x2b2ac00, 0xc003a5d000, 0xa7, 0x400, 0xc000982978, 0x19d1920, 0xc00095d650)
2021-05-27T18:51:12.9576360Z 	external/org_uber_go_zap/zaptest/logger.go:130 +0x123
2021-05-27T18:51:12.9577250Z go.uber.org/zap/zapcore.(*ioCore).Write(0xc00270ba10, 0x2, 0xc0241868341d8dbb, 0x8737a9917a, 0x2b2ac80, 0xc002276700, 0x10, 0x1c6a171, 0x11, 0x0, ...)
2021-05-27T18:51:12.9578091Z 	external/org_uber_go_zap/zapcore/core.go:90 +0x10c
2021-05-27T18:51:12.9578856Z go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc00070a210, 0xc0032722c0, 0x1, 0x1)
2021-05-27T18:51:12.9579610Z 	external/org_uber_go_zap/zapcore/entry.go:216 +0x12d
2021-05-27T18:51:12.9581228Z github.com/go-logr/zapr.(*zapLogger).Error(0xc0028edc00, 0x1eaf380, 0xc000058290, 0x1c6a171, 0x11, 0x0, 0x0, 0x0)
2021-05-27T18:51:12.9582023Z 	external/com_github_go_logr_zapr/zapr.go:133 +0x258
2021-05-27T18:51:12.9584027Z github.com/cockroachdb/cockroach-operator/pkg/update.makeIsCRBPodIsRunningNewVersionFunction.func1(0xc0018bc780, 0x0, 0x1ef6c08, 0xc0028edc00, 0xd1c1944705, 0xc000c09e00)
2021-05-27T18:51:12.9585881Z 	pkg/update/update_cockroach_version_common.go:89 +0xa04
2021-05-27T18:51:12.9587759Z github.com/cockroachdb/cockroach-operator/pkg/update.waitUntilPerPodVerificationFuncVerifies.func1(0xc003e0e5c8, 0xc003e0e598)
2021-05-27T18:51:12.9589420Z 	pkg/update/update.go:234 +0x49
2021-05-27T18:51:12.9590154Z github.com/cenkalti/backoff.RetryNotify(0xc003e0e628, 0x1ebbdd8, 0xc0002c0c00, 0x0, 0x0, 0x0)
2021-05-27T18:51:12.9591003Z 	external/com_github_cenkalti_backoff/retry.go:37 +0x1e8
2021-05-27T18:51:12.9591639Z github.com/cenkalti/backoff.Retry(...)
2021-05-27T18:51:12.9592279Z 	external/com_github_cenkalti_backoff/retry.go:24
2021-05-27T18:51:12.9594183Z github.com/cockroachdb/cockroach-operator/pkg/update.waitUntilPerPodVerificationFuncVerifies(0xc0018bc780, 0xc000983560, 0x0, 0xc002f48450, 0x1ef6c08, 0xc0028edc00, 0x0, 0x0)
2021-05-27T18:51:12.9595585Z 	pkg/update/update.go:240 +0xc5
2021-05-27T18:51:12.9597738Z github.com/cockroachdb/cockroach-operator/pkg/update.PartitionedRollingUpdateStrategy.func1(0xc0018bc780, 0xc002f48450, 0x1ef6c08, 0xc0028edc00, 0x4, 0x0, 0x0)
2021-05-27T18:51:12.9599141Z 	pkg/update/update.go:207 +0x4c5
2021-05-27T18:51:12.9602547Z github.com/cockroachdb/cockroach-operator/pkg/update.UpdateClusterRegionStatefulSet(0x1eec618, 0xc00270b5f0, 0x1f13c98, 0xc00311fa20, 0xc0022766e4, 0x4, 0xc002276700, 0x10, 0xc003e0e998, 0xc000983620, ...)
2021-05-27T18:51:12.9604168Z 	pkg/update/update.go:149 +0x2c2
2021-05-27T18:51:12.9605789Z github.com/cockroachdb/cockroach-operator/pkg/update.updateClusterStatefulSets(0x1eec618, 0xc00270b5f0, 0xc00241d900, 0xc002eee300, 0xc003e0e998, 0x1ef6c08, 0xc00271c450, 0x0, 0x0)
2021-05-27T18:51:12.9607085Z 	pkg/update/update_cockroach_version.go:132 +0x11f
2021-05-27T18:51:12.9608738Z github.com/cockroachdb/cockroach-operator/pkg/update.UpdateClusterCockroachVersion(0x1eec618, 0xc00270b5f0, 0xc00241d900, 0xc002eee300, 0x1ef6c08, 0xc00271c450, 0x9, 0xc003e02d00)
2021-05-27T18:51:12.9610064Z 	pkg/update/update_cockroach_version.go:117 +0x67a
2021-05-27T18:51:12.9611577Z github.com/cockroachdb/cockroach-operator/pkg/actor.(*partitionedUpdate).Act(0xc000d40e70, 0x1eec618, 0xc00270b5f0, 0xc00166b300, 0x0, 0x0)
2021-05-27T18:51:12.9612932Z 	pkg/actor/partitioned_update.go:224 +0x119c
2021-05-27T18:51:12.9615211Z github.com/cockroachdb/cockroach-operator/pkg/controller.(*ClusterReconciler).Reconcile(0xc002fb2a40, 0x1eec618, 0xc00270b5f0, 0xc003e20a50, 0x10, 0xc003e20a34, 0x4, 0xc00270b300, 0x0, 0x0, ...)
2021-05-27T18:51:12.9619385Z 	pkg/controller/cluster_controller.go:130 +0x1027
2021-05-27T18:51:12.9620982Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc002f94aa0, 0x1eec570, 0xc00310a140, 0x1aaf860, 0xc000c3c2a0)
2021-05-27T18:51:12.9622477Z 	external/io_k8s_sigs_controller_runtime/pkg/internal/controller/controller.go:297 +0x30d
2021-05-27T18:51:12.9623988Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc002f94aa0, 0x1eec570, 0xc00310a140, 0x0)
2021-05-27T18:51:12.9625262Z 	external/io_k8s_sigs_controller_runtime/pkg/internal/controller/controller.go:252 +0x205
2021-05-27T18:51:12.9626851Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2(0x1eec570, 0xc00310a140)
2021-05-27T18:51:12.9628130Z 	external/io_k8s_sigs_controller_runtime/pkg/internal/controller/controller.go:215 +0x4a
2021-05-27T18:51:12.9629249Z k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
2021-05-27T18:51:12.9630766Z 	external/io_k8s_apimachinery/pkg/util/wait/wait.go:185 +0x37
2021-05-27T18:51:12.9631587Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc003033f50)
2021-05-27T18:51:12.9632388Z 	external/io_k8s_apimachinery/pkg/util/wait/wait.go:155 +0x5f
2021-05-27T18:51:12.9633276Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc003e0ff50, 0x1eb1620, 0xc000d497d0, 0xc00310a101, 0xc002921200)
2021-05-27T18:51:12.9634380Z 	external/io_k8s_apimachinery/pkg/util/wait/wait.go:156 +0x9b
2021-05-27T18:51:12.9635195Z k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc003033f50, 0x3b9aca00, 0x0, 0x1140b01, 0xc002921200)
2021-05-27T18:51:12.9636036Z 	external/io_k8s_apimachinery/pkg/util/wait/wait.go:133 +0x98
2021-05-27T18:51:12.9637652Z k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext(0x1eec570, 0xc00310a140, 0xc0006d8800, 0x3b9aca00, 0x0, 0x1)
2021-05-27T18:51:12.9639419Z 	external/io_k8s_apimachinery/pkg/util/wait/wait.go:185 +0xa6
2021-05-27T18:51:12.9640302Z k8s.io/apimachinery/pkg/util/wait.UntilWithContext(0x1eec570, 0xc00310a140, 0xc0006d8800, 0x3b9aca00)
2021-05-27T18:51:12.9641213Z 	external/io_k8s_apimachinery/pkg/util/wait/wait.go:99 +0x57
2021-05-27T18:51:12.9643282Z created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
2021-05-27T18:51:12.9644519Z 	external/io_k8s_sigs_controller_runtime/pkg/internal/controller/controller.go:212 +0x40d
2021-05-27T18:51:12.9646875Z panic: Log in goroutine after TestUpgradesMajorVersion20_1To20_2 has completed [recovered]
2021-05-27T18:51:12.9648720Z 	panic: Log in goroutine after TestUpgradesMajorVersion20_1To20_2 has completed
2021-05-27T18:51:12.9649535Z 
2021-05-27T18:51:12.9650274Z goroutine 3016 [running]:
2021-05-27T18:51:12.9650935Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
2021-05-27T18:51:12.9651912Z 	external/io_k8s_apimachinery/pkg/util/runtime/runtime.go:55 +0x109
2021-05-27T18:51:12.9652507Z panic(0x19d1920, 0xc00095d670)
2021-05-27T18:51:12.9653050Z 	GOROOT/src/runtime/panic.go:965 +0x1b9
2021-05-27T18:51:12.9653658Z testing.(*common).logDepth(0xc00252af00, 0xc003b4c9a0, 0xa6, 0x3)
2021-05-27T18:51:12.9654284Z 	GOROOT/src/testing/testing.go:765 +0x5df
2021-05-27T18:51:12.9654790Z testing.(*common).log(...)
2021-05-27T18:51:12.9655267Z 	GOROOT/src/testing/testing.go:747
2021-05-27T18:51:12.9655881Z testing.(*common).Logf(0xc00252af00, 0x1c5639e, 0x2, 0xc00095d660, 0x1, 0x1)
2021-05-27T18:51:12.9656485Z 	GOROOT/src/testing/testing.go:793 +0x7e
2021-05-27T18:51:12.9657444Z go.uber.org/zap/zaptest.testingWriter.Write(0x1ef7598, 0xc00252af00, 0x2b2ac00, 0xc003a5d000, 0xa7, 0x400, 0xc000982978, 0x19d1920, 0xc00095d650)
2021-05-27T18:51:12.9658587Z 	external/org_uber_go_zap/zaptest/logger.go:130 +0x123
2021-05-27T18:51:12.9659468Z go.uber.org/zap/zapcore.(*ioCore).Write(0xc00270ba10, 0x2, 0xc0241868341d8dbb, 0x8737a9917a, 0x2b2ac80, 0xc002276700, 0x10, 0x1c6a171, 0x11, 0x0, ...)
2021-05-27T18:51:12.9660382Z 	external/org_uber_go_zap/zapcore/core.go:90 +0x10c
2021-05-27T18:51:12.9661098Z go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc00070a210, 0xc0032722c0, 0x1, 0x1)
2021-05-27T18:51:12.9661932Z 	external/org_uber_go_zap/zapcore/entry.go:216 +0x12d
2021-05-27T18:51:12.9663069Z github.com/go-logr/zapr.(*zapLogger).Error(0xc0028edc00, 0x1eaf380, 0xc000058290, 0x1c6a171, 0x11, 0x0, 0x0, 0x0)
2021-05-27T18:51:12.9663851Z 	external/com_github_go_logr_zapr/zapr.go:133 +0x258
2021-05-27T18:51:12.9665926Z github.com/cockroachdb/cockroach-operator/pkg/update.makeIsCRBPodIsRunningNewVersionFunction.func1(0xc0018bc780, 0x0, 0x1ef6c08, 0xc0028edc00, 0xd1c1944705, 0xc000c09e00)
2021-05-27T18:51:12.9667625Z 	pkg/update/update_cockroach_version_common.go:89 +0xa04
2021-05-27T18:51:12.9669615Z github.com/cockroachdb/cockroach-operator/pkg/update.waitUntilPerPodVerificationFuncVerifies.func1(0xc003e0e5c8, 0xc003e0e598)
2021-05-27T18:51:12.9671020Z 	pkg/update/update.go:234 +0x49
2021-05-27T18:51:12.9671744Z github.com/cenkalti/backoff.RetryNotify(0xc003e0e628, 0x1ebbdd8, 0xc0002c0c00, 0x0, 0x0, 0x0)
2021-05-27T18:51:12.9672548Z 	external/com_github_cenkalti_backoff/retry.go:37 +0x1e8
2021-05-27T18:51:12.9673170Z github.com/cenkalti/backoff.Retry(...)
2021-05-27T18:51:12.9673771Z 	external/com_github_cenkalti_backoff/retry.go:24
2021-05-27T18:51:12.9675516Z github.com/cockroachdb/cockroach-operator/pkg/update.waitUntilPerPodVerificationFuncVerifies(0xc0018bc780, 0xc000983560, 0x0, 0xc002f48450, 0x1ef6c08, 0xc0028edc00, 0x0, 0x0)
2021-05-27T18:51:12.9680806Z 	pkg/update/update.go:240 +0xc5
2021-05-27T18:51:12.9682589Z github.com/cockroachdb/cockroach-operator/pkg/update.PartitionedRollingUpdateStrategy.func1(0xc0018bc780, 0xc002f48450, 0x1ef6c08, 0xc0028edc00, 0x4, 0x0, 0x0)
2021-05-27T18:51:12.9683974Z 	pkg/update/update.go:207 +0x4c5
2021-05-27T18:51:12.9685604Z github.com/cockroachdb/cockroach-operator/pkg/update.UpdateClusterRegionStatefulSet(0x1eec618, 0xc00270b5f0, 0x1f13c98, 0xc00311fa20, 0xc0022766e4, 0x4, 0xc002276700, 0x10, 0xc003e0e998, 0xc000983620, ...)
2021-05-27T18:51:12.9686912Z 	pkg/update/update.go:149 +0x2c2
2021-05-27T18:51:12.9688639Z github.com/cockroachdb/cockroach-operator/pkg/update.updateClusterStatefulSets(0x1eec618, 0xc00270b5f0, 0xc00241d900, 0xc002eee300, 0xc003e0e998, 0x1ef6c08, 0xc00271c450, 0x0, 0x0)
2021-05-27T18:51:12.9690326Z 	pkg/update/update_cockroach_version.go:132 +0x11f
2021-05-27T18:51:12.9692175Z github.com/cockroachdb/cockroach-operator/pkg/update.UpdateClusterCockroachVersion(0x1eec618, 0xc00270b5f0, 0xc00241d900, 0xc002eee300, 0x1ef6c08, 0xc00271c450, 0x9, 0xc003e02d00)
2021-05-27T18:51:12.9693497Z 	pkg/update/update_cockroach_version.go:117 +0x67a
2021-05-27T18:51:12.9694788Z github.com/cockroachdb/cockroach-operator/pkg/actor.(*partitionedUpdate).Act(0xc000d40e70, 0x1eec618, 0xc00270b5f0, 0xc00166b300, 0x0, 0x0)
2021-05-27T18:51:12.9695768Z 	pkg/actor/partitioned_update.go:224 +0x119c
2021-05-27T18:51:12.9697240Z github.com/cockroachdb/cockroach-operator/pkg/controller.(*ClusterReconciler).Reconcile(0xc002fb2a40, 0x1eec618, 0xc00270b5f0, 0xc003e20a50, 0x10, 0xc003e20a34, 0x4, 0xc00270b300, 0x0, 0x0, ...)
2021-05-27T18:51:12.9698422Z 	pkg/controller/cluster_controller.go:130 +0x1027
2021-05-27T18:51:12.9699955Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc002f94aa0, 0x1eec570, 0xc00310a140, 0x1aaf860, 0xc000c3c2a0)
2021-05-27T18:51:12.9705807Z 	external/io_k8s_sigs_controller_runtime/pkg/internal/controller/controller.go:297 +0x30d
2021-05-27T18:51:12.9707724Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc002f94aa0, 0x1eec570, 0xc00310a140, 0x0)
2021-05-27T18:51:12.9709230Z 	external/io_k8s_sigs_controller_runtime/pkg/internal/controller/controller.go:252 +0x205
2021-05-27T18:51:12.9710847Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2(0x1eec570, 0xc00310a140)
2021-05-27T18:51:12.9711970Z 	external/io_k8s_sigs_controller_runtime/pkg/internal/controller/controller.go:215 +0x4a
2021-05-27T18:51:12.9713095Z k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
2021-05-27T18:51:12.9714192Z 	external/io_k8s_apimachinery/pkg/util/wait/wait.go:185 +0x37
2021-05-27T18:51:12.9715056Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc003033f50)
2021-05-27T18:51:12.9715889Z 	external/io_k8s_apimachinery/pkg/util/wait/wait.go:155 +0x5f
2021-05-27T18:51:12.9717517Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc003e0ff50, 0x1eb1620, 0xc000d497d0, 0xc00310a101, 0xc002921200)
2021-05-27T18:51:12.9718766Z 	external/io_k8s_apimachinery/pkg/util/wait/wait.go:156 +0x9b
2021-05-27T18:51:12.9719612Z k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc003033f50, 0x3b9aca00, 0x0, 0x1140b01, 0xc002921200)
2021-05-27T18:51:12.9720479Z 	external/io_k8s_apimachinery/pkg/util/wait/wait.go:133 +0x98
2021-05-27T18:51:12.9721487Z k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext(0x1eec570, 0xc00310a140, 0xc0006d8800, 0x3b9aca00, 0x0, 0x1)
2021-05-27T18:51:12.9722515Z 	external/io_k8s_apimachinery/pkg/util/wait/wait.go:185 +0xa6
2021-05-27T18:51:12.9723645Z k8s.io/apimachinery/pkg/util/wait.UntilWithContext(0x1eec570, 0xc00310a140, 0xc0006d8800, 0x3b9aca00)
2021-05-27T18:51:12.9724599Z 	external/io_k8s_apimachinery/pkg/util/wait/wait.go:99 +0x57
2021-05-27T18:51:12.9726022Z created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
2021-05-27T18:51:12.9727319Z 	external/io_k8s_sigs_controller_runtime/pkg/internal/controller/controller.go:212 +0x40d

Once in while.

I am uncertain why https://github.com/cockroachdb/cockroach-operator/blob/8c16730491e0705c8a3276b9039a298ef3ce722a/pkg/update/update_cockroach_version_common.go#L89

Would cause a panic

chrislovecnm avatar May 27 '21 18:05 chrislovecnm