Some tests timing out on helix

[radical]

trafficstars

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=804618&view=results Build error leg or test failing: Aspire.Hosting.Elasticsearch.Tests.WorkItemExecution Pull request: https://github.com/dotnet/aspire/pull/5243

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "Aborting test run: test run timeout of [0-9]+ milliseconds exceeded",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=804618 Error message validated: [Aborting test run: test run timeout of [0-9]+ milliseconds exceeded] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 9/11/2024 7:28:44 PM UTC

Report

Build	Definition	Test	Pull Request
827907	dotnet/aspire	Aspire.Pomelo.EntityFrameworkCore.MySql.Tests-net9.0.WorkItemExecution	dotnet/aspire#5932
827325	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	dotnet/aspire#6097
826882	dotnet/aspire	Aspire.Pomelo.EntityFrameworkCore.MySql.Tests-net9.0.WorkItemExecution	dotnet/aspire#5932
825649	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	dotnet/aspire#6045
823716	dotnet/aspire	Aspire.Elastic.Clients.Elasticsearch.Tests.WorkItemExecution	dotnet/aspire#6054
823203	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#6049
823148	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	dotnet/aspire#6049
822316	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5761
818636	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5870
818338	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5686
818082	dotnet/aspire	Aspire.Playground.Tests.ProjectSpecificTests.Aspire.Playground.Tests.ProjectSpecificTests.AzureFunctionsTest	dotnet/aspire#5905
816627	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5877
816570	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5670
816561	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5789
816004	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5809
816003	dotnet/aspire	Aspire.EndToEnd.Tests.IntegrationServicesTests.Aspire.EndToEnd.Tests.IntegrationServicesTests.VerifyCosmosComponentWorks(resourceName: efcosmos)
814886	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5771
814859	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	dotnet/aspire#5407
814709	dotnet/aspire	Aspire.EndToEnd.Tests.IntegrationServicesTests.Aspire.EndToEnd.Tests.IntegrationServicesTests.VerifyCosmosComponentWorks(resourceName: efcosmos)	dotnet/aspire#5807
814550	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5798
814785	dotnet/aspire	Aspire.EndToEnd.Tests.IntegrationServicesTests.Aspire.EndToEnd.Tests.IntegrationServicesTests.VerifyCosmosComponentWorks(resourceName: efcosmos)	dotnet/aspire#5771
814741	dotnet/aspire	Aspire.EndToEnd.Tests-cosmos.WorkItemExecution	dotnet/aspire#5380
814695	dotnet/aspire	Aspire.EndToEnd.Tests-cosmos.WorkItemExecution	dotnet/aspire#5802
814663	dotnet/aspire	Aspire.EndToEnd.Tests-cosmos.WorkItemExecution	dotnet/aspire#5779
814645	dotnet/aspire	Aspire.EndToEnd.Tests.IntegrationServicesTests.Aspire.EndToEnd.Tests.IntegrationServicesTests.VerifyCosmosComponentWorks(resourceName: efcosmos)	dotnet/aspire#5802
814590	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk
814133	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	dotnet/aspire#5621
814441	dotnet/aspire	Aspire.EndToEnd.Tests.IntegrationServicesTests.Aspire.EndToEnd.Tests.IntegrationServicesTests.VerifyCosmosComponentWorks(resourceName: efcosmos)	dotnet/aspire#5775
814167	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5806
814495	dotnet/aspire	Aspire.EndToEnd.Tests-cosmos.WorkItemExecution	dotnet/aspire#5817
814464	dotnet/aspire	Aspire.EndToEnd.Tests-cosmos.WorkItemExecution	dotnet/aspire#5802
814329	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	dotnet/aspire#5808
814294	dotnet/aspire	Aspire.EndToEnd.Tests.IntegrationServicesTests.Aspire.EndToEnd.Tests.IntegrationServicesTests.VerifyCosmosComponentWorks(resourceName: efcosmos)	dotnet/aspire#5798
813487	dotnet/aspire	Aspire.Hosting.SqlServer.Tests.WorkItemExecution	dotnet/aspire#5790
814079	dotnet/aspire	Aspire.EndToEnd.Tests-cosmos.WorkItemExecution	dotnet/aspire#5771
814077	dotnet/aspire	Aspire.EndToEnd.Tests-cosmos.WorkItemExecution	dotnet/aspire#5783
813969	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	dotnet/aspire#5380
813968	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	dotnet/aspire#5799
813909	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	dotnet/aspire#5804
813822	dotnet/aspire	Aspire.EndToEnd.Tests-cosmos.WorkItemExecution	dotnet/aspire#5798
813814	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution
813718	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5538
813696	dotnet/aspire	Aspire.EndToEnd.Tests.IntegrationServicesTests.Aspire.EndToEnd.Tests.IntegrationServicesTests.VerifyCosmosComponentWorks(resourceName: efcosmos)	dotnet/aspire#5380
813695	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	dotnet/aspire#5538
813691	dotnet/aspire	Aspire.EndToEnd.Tests-cosmos.WorkItemExecution
813655	dotnet/aspire	Aspire.EndToEnd.Tests.IntegrationServicesTests.Aspire.EndToEnd.Tests.IntegrationServicesTests.VerifyCosmosComponentWorks(resourceName: efcosmos)	dotnet/aspire#5775
813664	dotnet/aspire	Aspire.EndToEnd.Tests.IntegrationServicesTests.Aspire.EndToEnd.Tests.IntegrationServicesTests.VerifyCosmosComponentWorks(resourceName: efcosmos)
813595	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5686
813587	dotnet/aspire	Aspire.EndToEnd.Tests-cosmos.WorkItemExecution	dotnet/aspire#5695
813519	dotnet/aspire	Aspire.EndToEnd.Tests-cosmos.WorkItemExecution	dotnet/aspire#5407
813423	dotnet/aspire	Aspire.EndToEnd.Tests.IntegrationServicesTests.Aspire.EndToEnd.Tests.IntegrationServicesTests.VerifyCosmosComponentWorks(resourceName: efcosmos)	dotnet/aspire#5686
813350	dotnet/aspire	Aspire.EndToEnd.Tests-cosmos.WorkItemExecution	dotnet/aspire#5788
813237	dotnet/aspire	Aspire.EndToEnd.Tests-cosmos.WorkItemExecution	dotnet/aspire#5484
813130	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5759
813075	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5784
812765	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5770
812745	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution
812446	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	dotnet/aspire#5772
812252	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk
811559	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5751
811484	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5319
811474	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5751
811473	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution
811455	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5757
811412	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5621
811099	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk
811000	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5712
810838	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5087
810634	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5750
810372	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5729
810299	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5538
810218	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5538
810045	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution
809718	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	dotnet/aspire#5402
809705	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	dotnet/aspire#5621
809615	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	dotnet/aspire#5730
809584	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	dotnet/aspire#5725
809085	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution
808814	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	dotnet/aspire#5690
808485	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5690
808424	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	dotnet/aspire#5319
807745	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5538
808417	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5703
807822	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5592
807801	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5688
807605	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5511
807395	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution
807364	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5701
807341	dotnet/aspire	Aspire.Hosting.Milvus.Tests.MilvusFunctionalTests.Aspire.Hosting.Milvus.Tests.MilvusFunctionalTests.WithDataShouldPersistStateBetweenUsages	dotnet/aspire#5701
807298	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	dotnet/aspire#5511
807292	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	dotnet/aspire#5699
807319	dotnet/aspire	Aspire.Hosting.SqlServer.Tests.SqlServerFunctionalTests.Aspire.Hosting.SqlServer.Tests.SqlServerFunctionalTests.WithDataShouldPersistStateBetweenUsages(useVolume: False)
807228	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5538
807198	dotnet/aspire	Aspire.Hosting.Containers.Tests.WithDockerfileTests.Aspire.Hosting.Containers.Tests.WithDockerfileTests.WithBuildSecretPopulatesSecretFilesCorrectly	dotnet/aspire#5511
807197	dotnet/aspire	Aspire.Hosting.MongoDB.Tests.MongoDbFunctionalTests.Aspire.Hosting.MongoDB.Tests.MongoDbFunctionalTests.VerifyWithInitBindMount	dotnet/aspire#5690
807190	dotnet/aspire	Aspire.Hosting.MongoDB.Tests.MongoDbFunctionalTests.Aspire.Hosting.MongoDB.Tests.MongoDbFunctionalTests.VerifyWithInitBindMount	dotnet/aspire#5592
807177	dotnet/aspire	Aspire.EndToEnd.Tests.IntegrationServicesTests.Aspire.EndToEnd.Tests.IntegrationServicesTests.VerifyCosmosComponentWorks(resourceName: efcosmos)	dotnet/aspire#5697
807139	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5538
807107	dotnet/aspire	Aspire.Playground.Tests.AppHostTests.Aspire.Playground.Tests.AppHostTests.TestEndpointsReturnOk	dotnet/aspire#5575
807124	dotnet/aspire	Aspire.Playground.Tests.WorkItemExecution	dotnet/aspire#5695
Displaying 100 of 139 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
3	8	139

[radical]

cc @eerhardt @sebastienros

One of the tests timing out is Aspire.Hosting.Elasticsearch.Tests - log.

Why is this timing out after 10minutes on helix when all the tests combined didn't take that long on the build machine?

• One reason could be that the helix agents are under-powered compared to the build machines. And these tests start up a new app per xunit test in ElasticsearchFunctionalTests.

[radical]

I chose a broader error message to match against, so we can get any tests that are timing out on helix now.

[mitchdenny]

I am still hitting this today on this PR: https://github.com/dotnet/aspire/pull/5223

[radical]

I am still hitting this today on this PR: #5223

I'll bump the timeouts. We are hitting new ones as we moved new tests to helix last week.

[radical]

Aspire.Hosting.Elasticsearch.Tests - I think this is timing out because each of the functional tests starts a new app, and the elasticsearch containers take a few minutes to start up, causing the time taken for all the tests to be more than 10 minutes.

[mitchdenny]

This just happened for mongo in the playground tests. I was briefly able to repro a timeout locally as well, but when I stopped the test and retried it worked fine (multiple times). When it was timing out I noticed that mongo express didn't have an external port allocated. Make me think that Docker failed to forward the port into the container. This would explain it waiting forever to be able to connect to the container via the endpoint.

[joperezr]

@radical Is there any additional logging that we can add to help us narrow down what is going on here? It would be good if we can make this issue somehow actionable.

[radical]

This hasn't been hit recently, and likely been better with improved timeouts too. Closing.