TestSubscribeRotations unit test fails in CI

[shashankram]

Bug description: The test TestSubscribeRotations fails in the CI. Refer to the CI run https://github.com/openservicemesh/osm/runs/7549446959?check_suite_focus=true.

2022-07-27T21:47:59.5064137Z [90m9:47PM[0m [35mTRC[0m Certificate found in cache SerialNumber= [36mcomponent=[0mcertificate [36mfile=[0mmanager.go:249
2022-07-27T21:47:59.5064897Z [90m9:47PM[0m [32mINF[0m Cert foo.fake.domain.com should be rotated; expires in -1h0m5.000665606s; renewBefore is 31s [36mcomponent=[0mcertificate [36mfile=[0mmanager.go:177
2022-07-27T21:47:59.5065616Z [90m9:47PM[0m [35mTRC[0m It took 19.7µs to issue certificate with SerialNumber= [36mcomponent=[0mcertificate [36mfile=[0mmanager.go:310
2022-07-27T21:47:59.5066301Z [90m9:47PM[0m [33mDBG[0m Rotated certificate (old SerialNumber=) with new SerialNumber= [36mcomponent=[0mcertificate [36mfile=[0mmanager.go:316
2022-07-27T21:47:59.5066689Z coverage: 71.1% of statements
2022-07-27T21:47:59.5066957Z panic: test timed out after 2m0s
2022-07-27T21:47:59.5067112Z 
2022-07-27T21:47:59.5067218Z goroutine 52 [running]:
2022-07-27T21:47:59.5067473Z testing.(*M).startAlarm.func1()
2022-07-27T21:47:59.5067839Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/testing/testing.go:1788 +0xbb
2022-07-27T21:47:59.5068120Z created by time.goFunc
2022-07-27T21:47:59.5068449Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/time/sleep.go:180 +0x4a
2022-07-27T21:47:59.5068639Z 
2022-07-27T21:47:59.5068748Z goroutine 1 [chan receive]:
2022-07-27T21:47:59.5069046Z testing.(*T).Run(0xc0001a5ba0, {0x12582f0, 0x16}, 0x12bbe08)
2022-07-27T21:47:59.5069417Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/testing/testing.go:1307 +0x752
2022-07-27T21:47:59.5069725Z testing.runTests.func1(0x0)
2022-07-27T21:47:59.5070067Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/testing/testing.go:1598 +0x9a
2022-07-27T21:47:59.5070390Z testing.tRunner(0xc0001a5ba0, 0xc0000fbbc0)
2022-07-27T21:47:59.5070751Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/testing/testing.go:1259 +0x230
2022-07-27T21:47:59.5071124Z testing.runTests(0xc000194780, {0x1aff800, 0x9, 0x9}, {0x0, 0xc0001b5ac0, 0x1b222c0})
2022-07-27T21:47:59.5071518Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/testing/testing.go:1596 +0x7cb
2022-07-27T21:47:59.5071817Z testing.(*M).Run(0xc000194780)
2022-07-27T21:47:59.5072147Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/testing/testing.go:1504 +0x9d2
2022-07-27T21:47:59.5072424Z main.main()
2022-07-27T21:47:59.5072659Z 	_testmain.go:119 +0x325
2022-07-27T21:47:59.5072885Z 
2022-07-27T21:47:59.5072996Z goroutine 19 [chan receive]:
2022-07-27T21:47:59.5073269Z k8s.io/klog/v2.(*loggingT).flushDaemon(0x0)
2022-07-27T21:47:59.5073601Z 	/home/runner/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:1181 +0x8b
2022-07-27T21:47:59.5073891Z created by k8s.io/klog/v2.init.0
2022-07-27T21:47:59.5074222Z 	/home/runner/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:420 +0x1c5
2022-07-27T21:47:59.5074404Z 
2022-07-27T21:47:59.5074515Z goroutine 11 [chan receive]:
2022-07-27T21:47:59.5074808Z github.com/cskr/pubsub.(*PubSub).start(0xc000189010)
2022-07-27T21:47:59.5075238Z 	/home/runner/go/pkg/mod/github.com/cskr/[email protected]/pubsub.go:130 +0x1a5
2022-07-27T21:47:59.5075626Z created by github.com/cskr/pubsub.New
2022-07-27T21:47:59.5075981Z 	/home/runner/go/pkg/mod/github.com/cskr/[email protected]/pubsub.go:44 +0xfa
2022-07-27T21:47:59.5076220Z 
2022-07-27T21:47:59.5076332Z goroutine 37 [chan receive]:
2022-07-27T21:47:59.5076624Z github.com/cskr/pubsub.(*PubSub).start(0xc000410760)
2022-07-27T21:47:59.5077010Z 	/home/runner/go/pkg/mod/github.com/cskr/[email protected]/pubsub.go:130 +0x1a5
2022-07-27T21:47:59.5077812Z created by github.com/cskr/pubsub.New
2022-07-27T21:47:59.5078207Z 	/home/runner/go/pkg/mod/github.com/cskr/[email protected]/pubsub.go:44 +0xfa
2022-07-27T21:47:59.5078405Z 
2022-07-27T21:47:59.5078514Z goroutine 39 [chan receive]:
2022-07-27T21:47:59.5078806Z github.com/cskr/pubsub.(*PubSub).start(0xc000411050)
2022-07-27T21:47:59.5079190Z 	/home/runner/go/pkg/mod/github.com/cskr/[email protected]/pubsub.go:130 +0x1a5
2022-07-27T21:47:59.5079516Z created by github.com/cskr/pubsub.New
2022-07-27T21:47:59.5079876Z 	/home/runner/go/pkg/mod/github.com/cskr/[email protected]/pubsub.go:44 +0xfa
2022-07-27T21:47:59.5080082Z 
2022-07-27T21:47:59.5080188Z goroutine 33 [chan send]:
2022-07-27T21:47:59.5080444Z github.com/cskr/pubsub.(*PubSub).Pub(...)
2022-07-27T21:47:59.5080805Z 	/home/runner/go/pkg/mod/github.com/cskr/[email protected]/pubsub.go:86
2022-07-27T21:47:59.5081300Z github.com/openservicemesh/osm/pkg/certificate.(*Manager).issueCertificate(0xc0002982a0, {0x124fd40, 0xfbe505}, {0x1249b4f, 0xc000221110}, {0xc0000a3a90, 0x0, 0xd})
2022-07-27T21:47:59.5081782Z 	/home/runner/work/osm/osm/pkg/certificate/manager.go:314 +0x805
2022-07-27T21:47:59.5082188Z github.com/openservicemesh/osm/pkg/certificate.(*Manager).IssueCertificate.func1()
2022-07-27T21:47:59.5082608Z 	/home/runner/work/osm/osm/pkg/certificate/manager.go:260 +0xb9
2022-07-27T21:47:59.5082996Z golang.org/x/sync/singleflight.(*Group).doCall.func2(0xc0000a385f, 0xc00019f0e0, 0xc0000a39f8)
2022-07-27T21:47:59.5083692Z 	/home/runner/go/pkg/mod/golang.org/x/[email protected]/singleflight/singleflight.go:193 +0xba
2022-07-27T21:47:59.5084157Z golang.org/x/sync/singleflight.(*Group).doCall(0xc0002982f0, 0xc00019f0e0, {0x124fd40, 0xd}, 0xc0003f8030)
2022-07-27T21:47:59.5084792Z 	/home/runner/go/pkg/mod/golang.org/x/[email protected]/singleflight/singleflight.go:195 +0x12c
2022-07-27T21:47:59.5085233Z golang.org/x/sync/singleflight.(*Group).Do(0xc0002982f0, {0x124fd40, 0xd}, 0x50000c0003f8030)
2022-07-27T21:47:59.5085839Z 	/home/runner/go/pkg/mod/golang.org/x/[email protected]/singleflight/singleflight.go:108 +0x20f
2022-07-27T21:47:59.5086362Z github.com/openservicemesh/osm/pkg/certificate.(*Manager).IssueCertificate(0xc0002982a0, {0x124fd40, 0xd}, {0x1249b4f, 0x7}, {0xc0000a3a90, 0x0, 0x0})
2022-07-27T21:47:59.5086816Z 	/home/runner/work/osm/osm/pkg/certificate/manager.go:259 +0x13c
2022-07-27T21:47:59.5087221Z github.com/openservicemesh/osm/pkg/certificate.(*Manager).checkAndRotate(0xc0002982a0)
2022-07-27T21:47:59.5087637Z 	/home/runner/work/osm/osm/pkg/certificate/manager.go:219 +0x355
2022-07-27T21:47:59.5088059Z github.com/openservicemesh/osm/pkg/certificate.TestSubscribeRotations(0xc0001a41a0)
2022-07-27T21:47:59.5088508Z 	/home/runner/work/osm/osm/pkg/certificate/manager_test.go:407 +0xcd3
2022-07-27T21:47:59.5088825Z testing.tRunner(0xc0001a41a0, 0x12bbe08)
2022-07-27T21:47:59.5089282Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/testing/testing.go:1259 +0x230
2022-07-27T21:47:59.5089579Z created by testing.(*T).Run
2022-07-27T21:47:59.5089921Z 	/opt/hostedtoolcache/go/1.17.12/x64/src/testing/testing.go:1306 +0x727
2022-07-27T21:47:59.5090114Z 
2022-07-27T21:47:59.5090217Z goroutine 15 [select]:
2022-07-27T21:47:59.5090579Z github.com/openservicemesh/osm/pkg/certificate.(*Manager).startRotationTicker.func1()
2022-07-27T21:47:59.5091009Z 	/home/runner/work/osm/osm/pkg/certificate/manager.go:43 +0x13a
2022-07-27T21:47:59.5091408Z created by github.com/openservicemesh/osm/pkg/certificate.(*Manager).startRotationTicker
2022-07-27T21:47:59.5091831Z 	/home/runner/work/osm/osm/pkg/certificate/manager.go:40 +0x139
2022-07-27T21:47:59.5092076Z 
2022-07-27T21:47:59.5092192Z goroutine 30 [chan receive]:
2022-07-27T21:47:59.5092487Z github.com/cskr/pubsub.(*PubSub).start(0xc000051b30)
2022-07-27T21:47:59.5092872Z 	/home/runner/go/pkg/mod/github.com/cskr/[email protected]/pubsub.go:130 +0x1a5
2022-07-27T21:47:59.5093203Z created by github.com/cskr/pubsub.New
2022-07-27T21:47:59.5093555Z 	/home/runner/go/pkg/mod/github.com/cskr/[email protected]/pubsub.go:44 +0xfa
2022-07-27T21:47:59.5093755Z 
2022-07-27T21:47:59.5093861Z goroutine 50 [chan send]:
2022-07-27T21:47:59.5094203Z github.com/cskr/pubsub.(*registry).send(0xc000177cb0, {0x124fd4d, 0xd}, {0x11cf0c0, 0xc0003f71e0})
2022-07-27T21:47:59.5094626Z 	/home/runner/go/pkg/mod/github.com/cskr/[email protected]/pubsub.go:205 +0x17f
2022-07-27T21:47:59.5094975Z github.com/cskr/pubsub.(*PubSub).start(0xc0003f4610)
2022-07-27T21:47:59.5095358Z 	/home/runner/go/pkg/mod/github.com/cskr/[email protected]/pubsub.go:158 +0x109e
2022-07-27T21:47:59.5095683Z created by github.com/cskr/pubsub.New
2022-07-27T21:47:59.5096036Z 	/home/runner/go/pkg/mod/github.com/cskr/[email protected]/pubsub.go:44 +0xfa
2022-07-27T21:47:59.5096236Z 
2022-07-27T21:47:59.5096345Z goroutine 51 [chan receive]:
2022-07-27T21:47:59.5096724Z github.com/openservicemesh/osm/pkg/certificate.TestSubscribeRotations.func2()
2022-07-27T21:47:59.5097176Z 	/home/runner/work/osm/osm/pkg/certificate/manager_test.go:397 +0x85
2022-07-27T21:47:59.5097592Z created by github.com/openservicemesh/osm/pkg/certificate.TestSubscribeRotations
2022-07-27T21:47:59.5098027Z 	/home/runner/work/osm/osm/pkg/certificate/manager_test.go:396 +0xa74
2022-07-27T21:47:59.5098380Z FAIL	github.com/openservicemesh/osm/pkg/certificate	120.077s
2022-07-27T21:47:59.5098721Z === RUN   TestGetCertificateFromKubernetes

Affected area (please mark with X where applicable):

• Certificate Management [X]
• Tests [X]

[github-actions[bot]]

Added default label size/needed. Please consider re-labeling this issue appropriately.

[github-actions[bot]]

Added default label kind/needed. Please consider re-labeling this issue appropriately.

[keithmattix]

@shashankram @steeling This appears to be fixed in CI; can we close this issue?