Failure in k8s-operator TestAPIs

[jcsp]

https://buildkite.com/redpanda/redpanda/builds/18834#018490fa-0cc6-4580-b087-ae5669dddab4

[joejulian]

@nicolaferraro What is this test? https://github.com/redpanda-data/redpanda/blob/28060714f66ec9c6bf31a8531f6dac255d6bd33d/src/go/k8s/controllers/redpanda/cluster_controller_configuration_test.go#L400

[dlex]

https://buildkite.com/redpanda/redpanda/builds/18931#0184a10c-4fcf-4920-a691-89afa2f23fca

[ztlpn]

https://buildkite.com/redpanda/redpanda/builds/18931#0184a10c-4fcf-4920-a691-89afa2f23fca

this is a different one btw:

[91m[1m• Failure in Spec Setup (BeforeEach) [0.001 seconds][0m

Console controller

[90m/var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-081b98b26548f5a25-1/redpanda/redpanda/src/go/k8s/controllers/redpanda/console_controller_test.go:48[0m

  [91m[1mWhen creating Console [BeforeEach][0m

  [90m/var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-081b98b26548f5a25-1/redpanda/redpanda/src/go/k8s/controllers/redpanda/console_controller_test.go:101[0m

    Should expose Console web app

    [90m/var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-081b98b26548f5a25-1/redpanda/redpanda/src/go/k8s/controllers/redpanda/console_controller_test.go:103[0m



    [91mExpected

        <*cache.ErrCacheNotStarted | 0x459fee0>: {}

    to equal

        <nil>: nil[0m



    /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-081b98b26548f5a25-1/redpanda/redpanda/src/go/k8s/controllers/redpanda/console_controller_test.go:68

[ztlpn]

Seems to fail pretty consistently on 22.3 now: https://buildkite.com/redpanda/redpanda/builds/19248#0184c528-7085-4cd6-a926-146b25b578c0

[ballard26]

Another instance of the issue https://buildkite.com/redpanda/redpanda/builds/20539#0185792e-8973-42ff-9e85-a9332cbd15fe

[dlex]

https://buildkite.com/redpanda/redpanda/builds/20589#01857e39-b97f-48dc-bde1-613d7597e84c

=== RUN   TestAPIs                                                                                                                                                                
Running Suite: Controller Suite                                                                                                                                                   
===============================                                                                                                                                                   
Random Seed: 1672860165                                                                                                                                                           
Will run 40 of 40 specs                                                                                                                                                           
                                                                                                                                                                                  
•••••••••••••••••••••••••••                                                                                                                                                       
------------------------------                                                                                                                                                    
• Failure [0.408 seconds]                                                                                                                                                         
RedPandaCluster configuration controller                                                                                                                                          
/var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-044e8dcaf8c5e76a4-1/redpanda/redpanda/src/go/k8s/controllers/redpanda/cluster_controller_configuration_test.go:37      
  When reconciling a cluster without centralized configuration                                                                                                                    
  /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-044e8dcaf8c5e76a4-1/redpanda/redpanda/src/go/k8s/controllers/redpanda/cluster_controller_configuration_test.go:375   
    Should behave like before [It]                                                                                                                                                
    /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-044e8dcaf8c5e76a4-1/redpanda/redpanda/src/go/k8s/controllers/redpanda/cluster_controller_configuration_test.go:376 
                                                                                                                                                                                  
    Failed after 0.000s.                                                                                                                                                          
    Expected                                                                                                                                                                      
        <int>: 1                                                                                                                                                                  
    to equal                                                                                                                                                                      
        <int>: 0                                                                                                                                                                  
                                                                                                                                                                                  
    /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-044e8dcaf8c5e76a4-1/redpanda/redpanda/src/go/k8s/controllers/redpanda/cluster_controller_configuration_test.go:400 

[joejulian]

Odd. Nothing was logged. It should have logged something if it touched the mock admin api.

[piyushredpanda]

@joejulian : what are the next steps here -- could you please advise? We cannot leave this failing as it has been very very frequent. What does it test? Can it be disabled?

[piyushredpanda]

@joejulian : I also see a PR open from December; can you please help us understand better?

[joejulian]

@piyushredpanda I couldn't make it fail locally even though I let this test run 250 times today. I do have a theory, that the above PR will address, that the operator is still reconciling a cluster from the previous test, causing unexpected changes to the mockAdminAPI. If that's true, then that PR will ensure that the deletion of the cluster from the previous test has been completed before starting the next test.

[piyushredpanda]

Awesome, many thanks for the explanation. FWIW, we've had similar shaped failures (previous test leaving bad state, etc) on the Core side; excited to see if this closes out!

[joejulian]

The backport of this fix is causing a consistent failure in v22.2.x so I'm reopening this until I get that solved.

[RafalKorepta]

Same issue in nightly

https://buildkite.com/redpanda/redpanda/builds/21490#0185c8a7-d829-4571-bddc-8480bbff8ed9/1127-1147

https://buildkite.com/redpanda/redpanda/builds/21488#0185c89d-18dd-42d8-9611-ee4eefc54d8e/1127-1147

[joejulian]

This is now fixed in dev branch.