IAM tests fail on EC2 clustered ducktape

[jcsp]

FAIL test: AWSRoleFetchTests.test_write (2/2 runs) failure at 2022-08-01T10:40:48.978Z: TimeoutError('Segments were not removed') in job https://buildkite.com/redpanda/vtools/builds/3077#01825803-32b3-4fee-8e12-243a0177833c failure at 2022-07-31T11:04:20.402Z: TimeoutError('Segments were not removed') in job https://buildkite.com/redpanda/vtools/builds/3073#018252da-dd82-4dac-bad9-ea37369ded6e FAIL test: STSRoleFetchTests.test_write (2/2 runs) failure at 2022-08-01T10:40:48.978Z: TimeoutError('Segments were not removed') in job https://buildkite.com/redpanda/vtools/builds/3077#01825803-32b3-4fee-8e12-243a0177833c failure at 2022-07-31T11:04:20.402Z: TimeoutError('Segments were not removed') in job https://buildkite.com/redpanda/vtools/builds/3073#018252da-dd82-4dac-bad9-ea37369ded6e FAIL test: ShortLivedCredentialsTests.test_short_lived_credentials (2/2 runs) failure at 2022-08-01T10:40:48.978Z: TimeoutError('Segments were not removed') in job https://buildkite.com/redpanda/vtools/builds/3077#01825803-32b3-4fee-8e12-243a0177833c failure at 2022-07-31T11:04:20.402Z: TimeoutError('Segments were not removed') in job https://buildkite.com/redpanda/vtools/builds/3073#018252da-dd82-4dac-bad9-ea37369ded6e

[jcsp]

I haven't looked in to these, don't know if they make sense when running against real S3?

[abhijat]

these tests use a mock service to retrieve and use the same credentials that the other tests use. they should work on real S3 just like other tests. will check the logs which are taking a long time to download for some reason

174 KB/s - 432 MB of 3.8 GB, 6 hours left

[BenPope]

Seen: https://buildkite.com/redpanda/vtools/builds/3139#01827b68-ebc9-4ff8-867f-d9461479bafe

[dotnwat]

@abhijat several examples of this here https://buildkite.com/redpanda/vtools/builds/3139#01827b68-ebc9-4ff8-867f-d9461479bafe

[abhijat]

sorry been busy with #5753 , will look into this one next

[abhijat]

I haven't looked in to these, don't know if they make sense when running against real S3?

@jcsp you were right, on looking further it seems like because the bundled mock server just returns static panda-user style credentials, whereas on CDT talking to real S3 we need the credentials from test context, the communication to s3 does not work within these tests.

We do not need to run these here, simply running against minio is good enough. I will look into removing these from running against S3.

[BenPope]

https://buildkite.com/redpanda/vtools/builds/3183#01828adc-074b-45e5-b224-1245d61997dc

Module: rptest.tests.e2e_iam_role_test
Class:  AWSRoleFetchTests
Method: test_write

Module: rptest.tests.e2e_iam_role_test
Class:  STSRoleFetchTests
Method: test_write

Module: rptest.tests.e2e_iam_role_test
Class:  ShortLivedCredentialsTests
Method: test_short_lived_credentials