dotnet
slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."
Build
https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results
Error
##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134
Build leg reported
vsos
Pull Request
No response
Known issue core information
Fill out the known issue JSON section by following the step by step documentation on how to create a known issue
{
"ErrorMessage" : "",
"BuildRetry": false,
"ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
"ExcludeConsoleLog": false
}
@dotnet/dnceng
Release Note Category
- [ ] Feature changes/additions
- [ ] Bug fixes
- [ ] Internal Infrastructure Improvements
Release Note Description
Additional information about the issue reported
No response
Known issue validation
Build: :mag_right: https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: :white_check_mark: Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC
Report
| Build | Definition | Step Name | Console log | Pull Request |
|---|---|---|---|---|
| 1211379 | dotnet/runtime | Libraries Test Run checked coreclr osx x64 Debug | Log | dotnet/runtime#120330 |
| 1211374 | dotnet/runtime | Libraries Test Run checked coreclr osx x64 Debug | Log | dotnet/runtime#121917 |
| 1211415 | dotnet/sdk | TestBuild: macOS (arm64) | Log | dotnet/sdk#50372 |
| 1211408 | dotnet/runtime | osx-x64 Release NativeAOT | Log | dotnet/runtime#121928 |
| 1211269 | dotnet/sdk | TestBuild: macOS (x64) | Log | dotnet/sdk#51869 |
| 1210799 | dotnet/sdk | TestBuild: macOS (arm64) | Log | dotnet/sdk#51847 |
| 1210723 | dotnet/runtime | maccatalyst-x64 Release AllSubsets_Mono | Log | |
| 2845772 | dotnet-runtime | coreclr Pri1 Runtime Tests Run osx arm64 checked | Log | |
| 2845675 | dotnet-runtime | coreclr Pri1 Runtime Tests Run osx arm64 checked | Log | |
| 2845328 | dotnet-runtime | Performance ios_scenarios iOSCoreCLR Interpreter iOSStripSymbols osx x64 perfiphone12mini net10.0 | Log | |
| 1210511 | dotnet/runtime | osx-x64 release Runtime_Release | Log | dotnet/runtime#121072 |
| 1210274 | dotnet/runtime | osx-x64 Release Libraries_CheckedCoreCLR | Log | |
| 1210152 | dotnet/runtime | osx-x64 Release NativeAOT | Log | dotnet/runtime#121866 |
| 1209844 | dotnet/sdk | TestBuild: macOS (arm64) | Log | dotnet/sdk#51826 |
| 2844644 | dotnet-maui | Build macOS | Log | |
| 2844209 | dotnet-runtime | Performance ios_scenarios iOSMono FullAOT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0 | Log | |
| 2844175 | dotnet-maui | MultiProject macOS | Log | |
| 1209304 | dotnet/runtime | osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit | Log | dotnet/runtime#120589 |
| 1209184 | dotnet/sdk | TestBuild: macOS (arm64) | Log | dotnet/sdk#51839 |
| 1209163 | dotnet/runtime | maccatalyst-x64 Release AllSubsets_Mono | Log | |
| 2843935 | dotnet-maui | Build macOS | Log | |
| 2843878 | dotnet-arcade-validation | Validate_Signing_MacOS Real_Signing | Log | |
| 1208835 | dotnet/sdk | TestBuild: macOS (arm64) | Log | dotnet/sdk#51808 |
| 1208603 | dotnet/runtime | ios-arm64 Release AllSubsets_CoreCLR | Log | dotnet/runtime#121305 |
| 2843542 | dotnet-maui | Build macOS | Log | |
| 1207980 | dotnet/runtime | tvossimulator-x64 Release AllSubsets_Mono_RuntimeTests | Log | |
| 2842910 | dotnet-efcore | macOS | Log | |
| 2842850 | dotnet-runtime | coreclr Pri1 Runtime Tests Run osx arm64 checked | Log | |
| 1207522 | dotnet/sdk | TestBuild: macOS (arm64) | Log | dotnet/sdk#51728 |
| 2842696 | dotnet-performance | Performance mlnet windows 22H2 arm64 Ampere 8.0 | Log | |
| 1207321 | dotnet/sdk | TestBuild: macOS (arm64) | Log | dotnet/sdk#51402 |
| 1207203 | dotnet/runtime | tvos-arm64 Release AllSubsets_Mono | Log | dotnet/runtime#121767 |
| 1207096 | dotnet/arcade | Linux Build_Debug | Log | |
| 1206878 | dotnet/sdk | TestBuild: macOS (arm64) | Log | dotnet/sdk#50864 |
| 1205930 | dotnet/sdk | TestBuild: macOS (arm64) | Log | dotnet/sdk#51784 |
| 1206747 | dotnet/sdk | TestBuild: macOS (arm64) | Log | dotnet/sdk#51765 |
| 2841799 | dotnet-performance | Performance powershell ubuntu 2204 x64 Tiger main | Log | |
| 2841784 | dotnet-maui | Build macOS | Log | |
| 1206320 | dotnet/runtime | coreclr Pri0 Runtime Tests Run osx arm64 checked | Log | dotnet/runtime#120688 |
| 1206319 | dotnet/runtime | maccatalyst-arm64 Release AllSubsets_Mono | Log | |
| 2841406 | dotnet-performance | Performance powershell ubuntu 2204 x64 Tiger main | Log | |
| 1206201 | dotnet/sdk | TestBuild: macOS (arm64) | Log | dotnet/sdk#51565 |
| 1206292 | dotnet/runtime | tvos-arm64 Release AllSubsets_NativeAOT | Log | |
| 1206252 | dotnet/runtime | Libraries Test Run release coreclr osx x64 Debug | Log | dotnet/runtime#121730 |
| 2838960 | dotnet-performance | Performance powershell windows Win11 x64 Viper 9.0 | Log | |
| 1206154 | dotnet/runtime | osx-arm64 Release NativeAOT | Log | dotnet/runtime#121724 |
| 2839775 | dotnet-performance | Performance powershell windows Win11 x64 Viper main | Log | |
| 1205992 | dotnet/runtime | coreclr Pri0 Runtime Tests Run osx x64 checked | Log | |
| 1206066 | dotnet/performance | Performance powershell ubuntu 2204 x64 Open main | Log | |
| 2840311 | dotnet-performance | Performance imagesharp ubuntu 2204 x64 Tiger main | Log | |
| 2841293 | dotnet-install-scripts | MacOSTests | ||
| 2841149 | dotnet-install-scripts | MacOSTests | Log | |
| 2835587 | dotnet-install-scripts | MacOSTests | Log | |
| 1205418 | dotnet/performance | Performance powershell ubuntu 2204 x64 Open 8.0 | Log | |
| 1205414 | dotnet/performance | Performance mlnet ubuntu 2204 x64 Open main | Log | dotnet/performance#5048 |
| 2840054 | dotnet-install-scripts | MacOSTests | Log | |
| 1204493 | dotnet/sdk | TestBuild: macOS (arm64) | Log | dotnet/sdk#51747 |
| 1204086 | dotnet/runtime | osx-x64 release Installer_Build_And_Test | Log | dotnet/runtime#121652 |
| 1204146 | dotnet/runtime | osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit | Log | dotnet/runtime#121656 |
| 1204083 | dotnet/runtime | maccatalyst-x64 Release AllSubsets_Mono | Log | dotnet/runtime#121653 |
| 1204029 | dotnet/runtime | coreclr Pri0 Runtime Tests Run osx x64 checked | Log | dotnet/runtime#119432 |
| 1204077 | dotnet/runtime | osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter | Log | |
| 2839504 | dotnet-runtime | ios-arm64 Release AllSubsets_NativeAOT | Log | #55295 |
| 1204014 | dotnet/runtime | osx-x64 release Installer_Build_And_Test | Log | dotnet/runtime#121650 |
| 1203955 | dotnet/runtime | coreclr Pri0 Runtime Tests Run osx x64 checked | Log | dotnet/runtime#121648 |
| 1203974 | dotnet/runtime | osx-arm64 Debug Installer_Build_And_Test | Log | dotnet/runtime#121457 |
| 2839448 | dotnet-runtime | Performance ios_scenarios iOSNativeAOT NativeAOT iOSStripSymbols osx x64 perfiphone12mini net10.0 | Log | |
| 1203995 | dotnet/runtime | ios-arm64 Release AllSubsets_NativeAOT | Log | dotnet/runtime#121649 |
| 1203929 | dotnet/runtime | coreclr Pri0 Runtime Tests Run osx x64 checked | Log | dotnet/runtime#121640 |
| 1203973 | dotnet/runtime | ios-arm64 Release AllSubsets_CoreCLR | Log | dotnet/runtime#121563 |
| 1203968 | dotnet/runtime | ios-arm64 Release AllSubsets_NativeAOT | Log | dotnet/runtime#121621 |
| 1203910 | dotnet/runtime | coreclr Pri0 Runtime Tests Run osx x64 checked | Log | dotnet/runtime#121646 |
| 1203882 | dotnet/runtime | osx-x64 release Installer_Build_And_Test | Log | dotnet/runtime#121643 |
| 2839349 | dotnet-runtime | Performance ios_scenarios iOSNativeAOT NativeAOT iOSStripSymbols osx x64 perfiphone12mini net10.0 | Log | |
| 1203863 | dotnet/runtime | coreclr Pri0 Runtime Tests Run osx arm64 checked | Log | |
| 1203906 | dotnet/runtime | maccatalyst-x64 Release AllSubsets_NativeAOT_RuntimeTests_AppSandbox | Log | |
| 1203804 | dotnet/runtime | coreclr Pri0 Runtime Tests Run osx x64 checked | Log | |
| 1203764 | dotnet/runtime | osx-arm64 Debug Installer_Build_And_Test | Log | dotnet/runtime#121637 |
| 1203851 | dotnet/runtime | ios-arm64 Release AllSubsets_Mono | Log | dotnet/runtime#121598 |
| 1203811 | dotnet/runtime | coreclr Pri0 Runtime Tests Run osx arm64 checked | Log | |
| 1203760 | dotnet/runtime | osx-x64 release Installer_Build_And_Test | Log | dotnet/runtime#121603 |
| 1203770 | dotnet/runtime | osx-arm64 Debug Installer_Build_And_Test | Log | dotnet/runtime#121639 |
| 1203814 | dotnet/runtime | maccatalyst-x64 Release AllSubsets_Mono_AppSandbox | Log | |
| 1203586 | dotnet/runtime | coreclr Pri0 Runtime Tests Run osx x64 checked | Log | dotnet/runtime#121508 |
| 1203607 | dotnet/runtime | coreclr Pri0 Runtime Tests Run osx arm64 checked | Log | dotnet/runtime#116203 |
| 1203597 | dotnet/runtime | osx-x64 release Installer_Build_And_Test | Log | dotnet/runtime#121633 |
| 1203641 | dotnet/runtime | ios-arm64 Release AllSubsets_CoreCLR | Log | dotnet/runtime#121417 |
| 1203539 | dotnet/runtime | osx-x64 release Installer_Build_And_Test | Log | dotnet/runtime#121634 |
| 1203410 | dotnet/runtime | coreclr Pri0 Runtime Tests Run osx x64 checked | Log | dotnet/runtime#121630 |
| 2839094 | dotnet-runtime | Performance ios_scenarios iOSNativeAOT NativeAOT iOSStripSymbols osx x64 perfiphone12mini net10.0 | Log | |
| 1203360 | dotnet/runtime | coreclr Pri0 Runtime Tests Run osx x64 checked | Log | dotnet/runtime#121341 |
| 1203337 | dotnet/runtime | osx-x64 release Installer_Build_And_Test | Log | dotnet/runtime#121625 |
| 1203340 | dotnet/runtime | coreclr Pri0 Runtime Tests Run osx x64 checked | Log | dotnet/runtime#121626 |
| 1203379 | dotnet/runtime | tvos-arm64 Release AllSubsets_Mono | Log | dotnet/runtime#121628 |
| 1203317 | dotnet/runtime | coreclr Pri0 Runtime Tests Run osx x64 checked | Log | dotnet/runtime#121622 |
| 1203305 | dotnet/runtime | coreclr Pri0 Runtime Tests Run osx x64 checked | Log | dotnet/runtime#120330 |
| 2839007 | dotnet-runtime | Performance ios_scenarios iOSNativeAOT NativeAOT iOSStripSymbols osx x64 perfiphone12mini net10.0 | Log | |
| 1203272 | dotnet/runtime | osx-arm64 Debug Installer_Build_And_Test | Log | dotnet/runtime#119203 |
| 1203263 | dotnet/runtime | osx-x64 release Installer_Build_And_Test | Log | dotnet/runtime#121621 |
| 1203192 | dotnet/runtime | coreclr Pri0 Runtime Tests Run osx x64 checked | Log | dotnet/runtime#121406 |
| Displaying 100 of 535 results |
Summary
| 24-Hour Hit Count | 7-Day Hit Count | 1-Month Count |
|---|---|---|
| 0 | 35 | 535 |
@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?
@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.
More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results
- On success:
Run tests in Helixstep reports:
Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
Build succeeded.
SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
2 Warning(s)
0 Error(s)
Time Elapsed 00:03:14.97
Killing running build processes...
Finishing: Run tests in Helix
ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5
- While on failure - timeout the same step gets stuck with:
Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.
ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf
NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.
Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.
Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.
Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.
We are still experiencing the problem.
Should we try to change the queue to osx.13.arm64?
@dotnet/dnceng
Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.
