aspire icon indicating copy to clipboard operation
aspire copied to clipboard

[playground-tests] Flaky `EventHubs` test in `TestEndpointsReturnOk`

Open radical opened this issue 1 year ago • 2 comments

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=818717 Build error leg or test failing: Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk Pull request: N/A

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": " Timed out waiting for the consumer message to be logged: 'Hello, from /test sent via producerClient'",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=818717 Error message validated: [ Timed out waiting for the consumer message to be logged: 'Hello, from /test sent via producerClient'] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 9/26/2024 7:43:41 PM UTC

Report

Build Definition Test Pull Request
825556 dotnet/aspire Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk dotnet/aspire#6045
824830 dotnet/aspire Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk dotnet/aspire#6066
823953 dotnet/aspire Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk dotnet/aspire#6057
822641 dotnet/aspire Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk dotnet/aspire#5999
821730 dotnet/aspire Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk
818717 dotnet/aspire Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 5 6

radical avatar Sep 26 '24 19:09 radical

On the last build the consumer shutdown with this exception:

      The HostOptions.BackgroundServiceExceptionBehavior is configured to StopHost. A BackgroundService has thrown an unhandled exception, and the IHost instance is stopping. To avoid this behavior, configure this to Ignore; however the BackgroundService will not be restarted.
      System.AggregateException: One or more errors occurred. (The messaging entity 'sb://emulatorns1.eventhubs.emulator.net/hub' could not be found. To know more visit https://aka.ms/sbResourceMgrExceptions.  (hub).  For troubleshooting information, see https://aka.ms/azsdk/net/eventhubs/exceptions/troubleshoot)
       ---> Azure.Messaging.EventHubs.EventHubsException(ResourceNotFound): The messaging entity 'sb://emulatorns1.eventhubs.emulator.net/hub' could not be found. To know more visit https://aka.ms/sbResourceMgrExceptions.  (hub).  For troubleshooting information, see https://aka.ms/azsdk/net/eventhubs/exceptions/troubleshoot
         at Azure.Messaging.EventHubs.AmqpError.ThrowIfErrorResponse(AmqpMessage response, String eventHubName)
         at Azure.Messaging.EventHubs.Amqp.AmqpClient.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy, CancellationToken cancellationToken)
         at Azure.Messaging.EventHubs.Amqp.AmqpClient.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy, CancellationToken cancellationToken)
         at Azure.Messaging.EventHubs.EventHubConnection.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy, CancellationToken cancellationToken)
         at Azure.Messaging.EventHubs.Primitives.EventProcessor`1.ValidateEventHubsConnectionAsync(CancellationToken cancellationToken)
         at Azure.Messaging.EventHubs.Primitives.EventProcessor`1.ValidateEventHubsConnectionAsync(CancellationToken cancellationToken)
         at Azure.Messaging.EventHubs.Primitives.EventProcessor`1.ValidateProcessingPreconditions(CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at Azure.Messaging.EventHubs.Primitives.EventProcessor`1.StartProcessingInternalAsync(Boolean async, CancellationToken cancellationToken)
         at Azure.Messaging.EventHubs.Primitives.EventProcessor`1.StartProcessingAsync(CancellationToken cancellationToken)
         at Azure.Messaging.EventHubs.EventProcessorClient.StartProcessingInternalAsync(Boolean async, CancellationToken cancellationToken)
         at Azure.Messaging.EventHubs.EventProcessorClient.StartProcessingAsync(CancellationToken cancellationToken)
         at EventHubsConsumer.Processor.ExecuteAsync(CancellationToken stoppingToken) in /datadisks/disk1/work/ABF7093F/w/B02B09B0/e/playground/AspireEventHub/EventHubsConsumer/Processor.cs:line 44
         at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)

.. specifically - Azure.Messaging.EventHubs.EventHubsException(ResourceNotFound): The messaging entity 'sb://emulatorns1.eventhubs.emulator.net/hub' could not be found. To know more visit https://aka.ms/sbResourceMgrExceptions. (hub). For troubleshooting information, see https://aka.ms/azsdk/net/eventhubs/exceptions/troubleshoot.

.. but the test keeps waiting for the message from the consumer, and times out.

We should fix the tests to also fail earlier, so the xunit test reports show the real cause, or closer to it.

cc @davidfowl @eerhardt

radical avatar Oct 02 '24 20:10 radical

There's a bug in the consumer, I think this is a resiliency problem.

davidfowl avatar Oct 05 '24 00:10 davidfowl