Flaky Azure ServiceBus tests

[timbussmann]

This issue captures a set of failing tests that failed multiple builds during March 9. and March 14 (almost all on March 14). After that time, all the builds have become stable without a clear indication of change. Some other flaky tests have been addressed and there not listed on this issue.

US Daylight savings time seems to have begun on March 13th. Clock adjustments, clock drifts/skews, or similar clock-related issues could potentially explain several of the failing tests.

Acceptance tests:

• When_auditing_message_with_TimeToBeReceived
  • test times out. It is suspected that the message with the TTBR header is discarded before it's being picked up before auditing the message, which is indicated by the context state that shows that the initial handler hasn't received the message at all.
• When_message_with_TimeToBeReceived_fails
  • test times out. It is suspected that the message with the TTBR header is discarded before it's being picked up before auditing the message, which is indicated by the context state that shows that the initial handler hasn't received the message at all.
• When_retrying_message_from_error_queue
  • The confirmed retry id doesn't match with the expected ID, the assertion looks like this: Error Message: String lengths are both 36. Strings differ at index 0. Expected: "61dfacbc-9218-4b0b-85d0-beab232dfefd" But was: "a10c4a04-dc8e-4747-a9dc-e17950fe7e82".
    • Each platform has it's own ASB namespace, so parallel matrix tests shouldn't interfere
    • Different platform (e.g. .NET Core 3.1 and .NET 6) tests per matrix configuration run against the same namespace but ASB tests should ignore messages from other test runs since messages are marked with a special header.
• When_blowing_up_just_after_dispatch
  • test timeout
• When_publishing.cs
  • test times out.
    • Log indicates that both subscribers have completed the subscribe operation. Only one subscriber receives the published event while the second subscriber doesn't receive the event until the test times out. A potential race condition in the ASB internal behavior when adding new subscriptions could potentially explain the behavior.
• When_disabling_serializer_type_inference.Should_not_deserialize_messages_without_types_header
  • test timeout
    • Log indicates that the message has been sent but not received within the tests time limit of 20s

Transport tests:

• When_cancelling_stop_on_message.cs
  • test timeout
• When_modifying_headers_before_on_error.cs
  • test timeout
• When_stopping_on_error.cs
  • test timeout
• When_message_is_available
  • test timeout

CommandLine tests:

• Create_queue_when_it_does_not_exist
  • Queue should be deleted before the create command is being executed, but the log output states Queue 'cli-queue' already exists, skipping creation
• Delete_queue_when_it_exists
  • VerifyQueueExists assertion fails
  • Log output indicates that the queue is being created multiple times (Queue name 'cli-queue', size '5GB', partitioned 'False' created ... Queue name 'cli-queue', size '5GB', partitioned 'False' created ... Queue 'cli-queue' already exists, skipping creation ... Queue name 'cli-queue', size '5GB', partitioned 'False' created ... Queue name 'cli-queue' deleted