Azure
Flaky Test Tracking
Describe the current behavior
Some of our tests are flaky; we've made improvements to our test infrastructure to improve things, but the tests need to be rerecorded for those improvements to kick in.
Describe the improvement
We should track tests that are flaky that haven't yet been updated, and treat them as a priority for rerecording
| Test | Flakes Observed | Recording Version |
|---|---|---|
| Test_Compute_VM_20220301_CRUD | 2024-08-28 | v1 |
| Test_DBForMySQL_FlexibleServer_20230630_CRUD | 2024-11-25 2024-11-19 |
v2 |
| Test_Dbforpostgresql_v1api20210601_CreationAndDeletion | 2025-04-30 | v1 |
| Test_DBForPostgreSQL_FlexibleServer_20230601Preview_CRUD | - | v1 |
| Test_DocumentDB_SQLDatabase_v20240815_CRUD | 2025-02-18 | v2 |
| Test_Insights_v1api_CreationAndDeletion | 2024-09-24 | v1 |
| Test_Keyvault_v1api20210401preview_CreationAndDeletion | 2025-03-06 | v1 |
| Test_Keyvault_v1api20230701_CreationAndDeletion | 2024-10-02 | v1 |
| Test_MariaDB_Server_CRUD | 2024-09-16 | v1 |
| Test_MissingSecret_ReturnsError_ReconcilesSuccessfullyWhenSecretAdded | 2024-12-13 | v1 |
| Test_Multitenant_SingleOperator_PerResourceCredential_MatchSubscriptionWithOwner | 2024-10-21 | - |
| Test_Owner_IsMutableIfNotSuccessfullyCreated | 2024-11-21 | v1 |
| Test_ReconcilePolicy_Skip_SkipsDelete | 2024-12-03 | v1 |
| TestRecorder_WhenRecordingAndRecordingDoesNotExist_MakesRecording | ??? | v2 |
| Test_Synapse_v1api_CreationAndDeletion | 2024-11-26 2024-10-01 2024-09-17 2024-09-05 2024-09-02 |
v1 |
| Test_UserSecretInDifferentNamespace_SecretNotFound | 2024-12-04 | v2 |
(Please update this table with new occurrences; list tests alphabetically, list newer occurrences first.)
Additional context Add any other context about the suggested improvement.
Samples tests sometimes flake too: Test_Samples_CreationAndDeletion/Test_Synapse_v1api_CreationAndDeletion I saw just now
Also seen: Test_Samples_CreationAndDeletion/Test_Keyvault_v1api20230701_CreationAndDeletion
I think I fixed the issue with Test_ReconcilePolicy_Skip_SkipsDelete in PR: #4820
[produce-markdown-summary] === RUN Test_DBForMySQL_FlexibleServer_20210501_CRUD/MySQL_Flexible_servers_AAD_Administrators_CRUD [produce-markdown-summary] kube_per_test_context.go:633: ------------------------------------------------------------------------------------ [produce-markdown-summary] kube_per_test_context.go:634: Creating Microsoft.ManagedIdentity/userAssignedIdentities resource asotest-mi-mlpeas [produce-markdown-summary] kube_per_test_context.go:635: ------------------------------------------------------------------------------------ [produce-markdown-summary] kube_per_test_context.go:480: [produce-markdown-summary] No future change is possible. Bailing out early after 10.202s. [produce-markdown-summary] Expected [produce-markdown-summary] <conditions.Condition>: { [produce-markdown-summary] Type: "Ready", [produce-markdown-summary] Status: "False", [produce-markdown-summary] Severity: "Error", [produce-markdown-summary] LastTransitionTime: { [produce-markdown-summary] Time: 2025-08-12T16:08:39Z, [produce-markdown-summary] }, [produce-markdown-summary] ObservedGeneration: 3, [produce-markdown-summary] Reason: "ReconciliationFailedPermanently", [produce-markdown-summary] Message: "cannot find go-vcr recording for request from test "Test_DBForMySQL_FlexibleServer_20210501_CRUD" (cassette: "recordings/Test_DBForMySQL_FlexibleServer_20210501_CRUD") (body mismatch): PUT https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/asotest-rg-cdficv/providers/Microsoft.DBforMySQL/flexibleServers/asotest-mysql-ghhjvf?api-version=2021-05-01\nShortest body diff: \n\n", [produce-markdown-summary] } [produce-markdown-summary] status to be "True", severity to be "".