CI Failure (ClientError: An error occurred (OperationAborted) when calling the CreateBucket operation: A conflicting conditional operation is currently in progress against this resource) in `SIAdminApiTest.test_topic_recovery_on_leader`

[bharathv]

https://buildkite.com/redpanda/vtools/builds/9617

Module: rptest.tests.shadow_indexing_admin_api_test
Class: SIAdminApiTest
Method: test_topic_recovery_on_leader

test_id:    SIAdminApiTest.test_topic_recovery_on_leader
status:     FAIL
run time:   12.997 seconds

ClientError('An error occurred (OperationAborted) when calling the CreateBucket operation: A conflicting conditional operation is currently in progress against this resource. Please try again.')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 182, in _do_run
    self.setup_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 260, in setup_test
    self.test.setup()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/test.py", line 91, in setup
    self.setUp()
  File "/home/ubuntu/redpanda/tests/rptest/tests/shadow_indexing_admin_api_test.py", line 69, in setUp
    super().setUp()
  File "/home/ubuntu/redpanda/tests/rptest/tests/redpanda_test.py", line 110, in setUp
    self.redpanda.start()
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1908, in start
    self.start_si()
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 2344, in start_si
    self.cloud_storage_client.create_bucket(
  File "/home/ubuntu/redpanda/tests/rptest/archival/s3_client.py", line 97, in create_bucket
    raise err 
  File "/home/ubuntu/redpanda/tests/rptest/archival/s3_client.py", line 89, in create_bucket
    res = self._cli.create_bucket(
  File "/usr/local/lib/python3.10/dist-packages/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.10/dist-packages/botocore/client.py", line 964, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (OperationAborted) when calling the CreateBucket operation: A conflicting conditional operation is currently in progress against this resource. Please try again.

JIRA Link: CORE-1467

[VladLazar]

This is strange...

The test doesn't even start a Redpanda node. It fails because the bucket creation races with the bucket deletion. Going to close this as infra noise, but please re-open if it happens again.

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/12441

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/12521

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/12994 *https://buildkite.com/redpanda/vtools/builds/13013 *https://buildkite.com/redpanda/vtools/builds/13061 *https://buildkite.com/redpanda/vtools/builds/13063

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/13191

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/12994 *https://buildkite.com/redpanda/vtools/builds/13013 *https://buildkite.com/redpanda/vtools/builds/13061 *https://buildkite.com/redpanda/vtools/builds/13063 *https://buildkite.com/redpanda/vtools/builds/13191

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/12994 *https://buildkite.com/redpanda/vtools/builds/13013 *https://buildkite.com/redpanda/vtools/builds/13061 *https://buildkite.com/redpanda/vtools/builds/13063 *https://buildkite.com/redpanda/vtools/builds/13191

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/12994 *https://buildkite.com/redpanda/vtools/builds/13013 *https://buildkite.com/redpanda/vtools/builds/13061 *https://buildkite.com/redpanda/vtools/builds/13063 *https://buildkite.com/redpanda/vtools/builds/13191

[travisdowns]

It looks to me like the new failures are due to hitting bucket quota:

	

ClientError('An error occurred (TooManyBuckets) when calling the CreateBucket operation: You have attempted to create more buckets than allowed')
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 182, in _do_run
    self.setup_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 267, in setup_test
    self.test.setup()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/test.py", line 91, in setup
    self.setUp()
  File "/home/ubuntu/redpanda/tests/rptest/tests/shadow_indexing_admin_api_test.py", line 69, in setUp
    super().setUp()
  File "/home/ubuntu/redpanda/tests/rptest/tests/redpanda_test.py", line 39, in setUp
    self.__redpanda.start()
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 2513, in start
    self.start_si()
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 2975, in start_si
    self.cloud_storage_client.create_bucket(
  File "/home/ubuntu/redpanda/tests/rptest/archival/s3_client.py", line 145, in create_bucket
    raise err
  File "/home/ubuntu/redpanda/tests/rptest/archival/s3_client.py", line 137, in create_bucket
    res = self._cli.create_bucket(
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/botocore/client.py", line 964, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (TooManyBuckets) when calling the CreateBucket operation: You have attempted to create more buckets than allowed

So it is not related to the original cause of this exception, but because the PT signature algo doesn't include the exception text in the signature, these get pointed here. So I'm closing and marking ci-ignore so PT opens a new bug with the right details.

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/16002