Azure/Teraform tests fail sporadically

[gregkalapos]

Error logs:

Error: [ERROR] Unable to List read-only keys for CosmosDB Account dotneta3700e91df3d4e38ac: documentdb.DatabaseAccountsClient#ListReadOnlyKeys: Failure sending request: StatusCode=504 -- Original Error: context deadline exceeded

I'm seeing those for a lot (all?) of CosmosDb tests, all tests below failed:

Initializing / Parallel / Linux / Test / MicrosoftAzureCosmosTests.Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureCosmosTests.Capture_Span_When_Create_Database – Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureCosmosTests
<1s
Initializing / Parallel / Linux / Test / MicrosoftAzureCosmosTests.Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureCosmosTests.Capture_Span_When_Delete_Database – Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureCosmosTests
<1s
Initializing / Parallel / Linux / Test / MicrosoftAzureCosmosTests.Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureCosmosTests.Capture_Span_When_Get_Database – Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureCosmosTests
<1s
Initializing / Parallel / Linux / Test / MicrosoftAzureCosmosTests.Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureCosmosTests.Capture_Span_When_List_Databases – Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureCosmosTests
<1s
Initializing / Parallel / Linux / Test / MicrosoftAzureCosmosTests.Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureCosmosTests.Capture_Span_When_Create_Collection – Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureCosmosTests
<1s
Initializing / Parallel / Linux / Test / MicrosoftAzureCosmosTests.Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureCosmosTests.Capture_Span_When_Delete_Collection – Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureCosmosTests
<1s
Initializing / Parallel / Linux / Test / MicrosoftAzureCosmosTests.Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureCosmosTests.Capture_Span_When_List_Collections – Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureCosmosTests
<1s
Initializing / Parallel / Linux / Test / MicrosoftAzureCosmosTests.Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureCosmosTests.Capture_Span_When_Create_Document – Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureCosmosTests
<1s
Initializing / Parallel / Linux / Test / MicrosoftAzureCosmosTests.Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureCosmosTests.Capture_Span_When_Upsert_Document – Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureCosmosTests
<1s
Initializing / Parallel / Linux / Test / MicrosoftAzureCosmosTests.Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureCosmosTests.Capture_Span_When_Delete_Document – Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureCosmosTests
<1s
Initializing / Parallel / Linux / Test / MicrosoftAzureCosmosTests.Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureCosmosTests.Capture_Span_When_Replace_Document – Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureCosmosTests
<1s
Initializing / Parallel / Linux / Test / MicrosoftAzureDocumentDbTests.Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureDocumentDbTests.Capture_Span_When_Create_Database – Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureDocumentDbTests
<1s
Initializing / Parallel / Linux / Test / MicrosoftAzureDocumentDbTests.Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureDocumentDbTests.Capture_Span_When_Delete_Database – Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureDocumentDbTests
<1s
Initializing / Parallel / Linux / Test / MicrosoftAzureDocumentDbTests.Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureDocumentDbTests.Capture_Span_When_Get_Database – Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureDocumentDbTests
<1s
Initializing / Parallel / Linux / Test / MicrosoftAzureDocumentDbTests.Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureDocumentDbTests.Capture_Span_When_List_Databases – Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureDocumentDbTests
<1s
Initializing / Parallel / Linux / Test / MicrosoftAzureDocumentDbTests.Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureDocumentDbTests.Capture_Span_When_Create_Collection – Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureDocumentDbTests
<1s
Initializing / Parallel / Linux / Test / MicrosoftAzureDocumentDbTests.Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureDocumentDbTests.Capture_Span_When_Delete_Collection – Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureDocumentDbTests
<1s
Initializing / Parallel / Linux / Test / MicrosoftAzureDocumentDbTests.Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureDocumentDbTests.Capture_Span_When_List_Collections – Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureDocumentDbTests
<1s
Initializing / Parallel / Linux / Test / MicrosoftAzureDocumentDbTests.Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureDocumentDbTests.Capture_Span_When_Create_Document – Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureDocumentDbTests
<1s
Initializing / Parallel / Linux / Test / MicrosoftAzureDocumentDbTests.Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureDocumentDbTests.Capture_Span_When_Delete_Document – Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureDocumentDbTests
<1s
Initializing / Parallel / Linux / Test / MicrosoftAzureDocumentDbTests.Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureDocumentDbTests.Capture_Span_When_Upsert_Document – Elastic.Apm.Azure.CosmosDb.Tests.MicrosoftAzureDocumentDbTests

Typically re-triggering the build helps and they are green the next time.

We need to find out why this happens and fix it to make those tests stable.