runtime icon indicating copy to clipboard operation
runtime copied to clipboard

Unable to pull image from mcr.microsoft.com

Open jozkee opened this issue 5 months ago • 11 comments

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1108670 Build error leg or test failing:

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "Unable to pull image mcr.microsoft.com/dotnet-buildtools/prereqs",
  "BuildRetry": true,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1108670 Result validation: :warning: Validation could not be done without an Azure DevOps build URL on the issue. Please add it to the "Build: :mag_right:" line. Validation performed at: 8/1/2025 8:10:46 AM UTC

Report

Build Definition Test Pull Request
1224778 dotnet/runtime NoWebcil-ST-Wasm.Build.NativeRebuild.Tests.ReferenceNewAssemblyRebuildTest.WorkItemExecution
1224677 dotnet/runtime Invariant.Tests.WorkItemExecution dotnet/runtime#122334
1224472 dotnet/runtime Microsoft.Extensions.Configuration.Json.Tests.WorkItemExecution dotnet/runtime#120823
1224127 dotnet/runtime System.Xml.Linq.Events.Tests.WorkItemExecution dotnet/runtime#122195
1221884 dotnet/runtime NoWebcil-ST-Wasm.Build.Tests.Blazor.SimpleRunTests.WorkItemExecution dotnet/runtime#122212
1223922 dotnet/runtime JIT.CodeGenBringUpTests_ro.WorkItemExecution dotnet/runtime#122296
1223696 dotnet/runtime System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http3.Public_Interop_ExactVersion_BufferContent_Success
1223692 dotnet/runtime System.Xml.Linq.Misc.Tests.WorkItemExecution
1223582 dotnet/runtime IcuAppLocal.Tests.WorkItemExecution
1223532 dotnet/runtime Microsoft.Extensions.Options.ConfigurationExtensions.SourceGeneration.Tests.WorkItemExecution dotnet/runtime#122160
1223380 dotnet/runtime Microsoft.Extensions.DependencyModel.Tests.WorkItemExecution dotnet/runtime#122276
1223348 dotnet/runtime NoWebcil-ST-Wasm.Build.Tests.WasmRunOutOfAppBundleTests.WorkItemExecution dotnet/runtime#122269
1223225 dotnet/runtime System.Xml.Linq.xNodeReader.Tests.WorkItemExecution
1223212 dotnet/runtime NoWebcil-ST-Wasm.Build.Tests.WasmTemplateTests.WorkItemExecution
1223096 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1223094 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1223042 dotnet/runtime Workloads-ST-Wasm.Build.NativeRebuild.Tests.OptimizationFlagChangeTests.WorkItemExecution dotnet/runtime#121853
1222836 dotnet/runtime threading_group2.WorkItemExecution
1222812 dotnet/runtime Workloads-ST-Wasm.Build.NativeRebuild.Tests.NoopNativeRebuildTest.WorkItemExecution dotnet/runtime#122256
1222803 dotnet/runtime WasmTestOnChrome-ST-System.Text.RegularExpressions.Tests.WorkItemExecution
1222720 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Intrinsics.Tests.WorkItemExecution dotnet/runtime#122258
1222690 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1222694 dotnet/runtime System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http3.Public_Interop_ExactVersion_BufferContent_Success
1222674 dotnet/runtime ComInterfaceGenerator.Tests.WorkItemExecution
1222664 dotnet/runtime Workloads-ST-Wasm.Build.Tests.Blazor.AppsettingsTests.WorkItemExecution
1222622 dotnet/runtime Workloads-ST-Wasm.Build.Tests.Blazor.DllImportTests.WorkItemExecution dotnet/runtime#122258
1222598 dotnet/runtime Workloads-ST-Wasm.Build.Tests.Blazor.DllImportTests.WorkItemExecution dotnet/runtime#121694
1222595 dotnet/runtime NoWebcil-ST-Wasm.Build.Tests.Blazor.NoopNativeRebuildTest.WorkItemExecution dotnet/runtime#121656
1222457 dotnet/runtime Wasm.Build.NativeRebuild.Tests.FlagsChangeRebuildTests.ExtraEmccFlagsSetButNoRealChange dotnet/runtime#121614
1222166 dotnet/runtime Regression_1.WorkItemExecution dotnet/runtime#122242
1221895 dotnet/runtime System.Collections.NonGeneric.Tests.WorkItemExecution
1221654 dotnet/runtime ComInterfaceGenerator.Unit.Tests.WorkItemExecution dotnet/runtime#122208
1221587 dotnet/runtime Workloads-ST-Wasm.Build.Tests.Blazor.SimpleRunTests.WorkItemExecution dotnet/runtime#120330
1221571 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1221487 dotnet/runtime System.Security.Cryptography.X509Certificates.Tests.CollectionTests.X509ChainElementCollection_CopyTo_NonZeroLowerBound_ThrowsIndexOutOfRangeException
1220963 dotnet/runtime chrome-DebuggerTests.AssignmentTests.WorkItemExecution dotnet/runtime#122199
1220544 dotnet/runtime JSImportGenerator.Unit.Tests.Compiles.ValidateGeneratedSourceOutput_AllAnnotatedParameters dotnet/runtime#121412
1220495 dotnet/runtime WasmTestOnChrome-ST-System.Net.Http.Functional.Tests.WorkItemExecution dotnet/runtime#121614
1220446 dotnet/runtime JIT_ro.WorkItemExecution dotnet/runtime#120330
1220399 dotnet/runtime NoWebcil-ST-Wasm.Build.Tests.PInvokeTableGeneratorTests.WorkItemExecution dotnet/runtime#121366
1220161 dotnet/runtime System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http3.Public_Interop_ExactVersion_BufferContent_Success
1220169 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1219965 dotnet/runtime Common.Tests.WorkItemExecution dotnet/runtime#122136
1219763 dotnet/runtime System.Net.Http.Functional.Tests.WorkItemExecution dotnet/runtime#122125
1219939 dotnet/runtime System.ComponentModel.TypeConverter.Tests.WorkItemExecution dotnet/runtime#122170
1219788 dotnet/runtime Microsoft.Extensions.Configuration.Tests.WorkItemExecution
1219771 dotnet/runtime profiler.transitions.WorkItemExecution dotnet/runtime#121343
1219748 dotnet/runtime threading_group2.WorkItemExecution
1219628 dotnet/runtime System.Xml.Linq.xNodeBuilder.Tests.WorkItemExecution dotnet/runtime#121973
1219633 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Intrinsics.Tests.WorkItemExecution
1219595 dotnet/runtime System.Net.Security.Tests.WorkItemExecution dotnet/runtime#121366
1219086 dotnet/runtime Microsoft.Bcl.Numerics.Tests.WorkItemExecution dotnet/runtime#121366
1218999 dotnet/runtime Workloads-ST-Wasm.Build.Tests.PreloadingTests.WorkItemExecution dotnet/runtime#122092
1218552 dotnet/runtime System.Runtime.Numerics.Tests.WorkItemExecution
1218559 dotnet/runtime System.Runtime.Numerics.Tests.WorkItemExecution
1217372 dotnet/runtime NoWebcil-ST-Wasm.Build.Tests.ConfigSrcTests.WorkItemExecution dotnet/runtime#122101
1217838 dotnet/runtime System.Text.RegularExpressions.Unit.Tests.WorkItemExecution dotnet/runtime#121158
1217746 dotnet/runtime System.Threading.Timer.Tests.WorkItemExecution
1217729 dotnet/runtime WasmTestOnChrome-ST-System.Text.RegularExpressions.Tests.WorkItemExecution
1217725 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Intrinsics.Tests.WorkItemExecution
1217160 dotnet/runtime Common.Tests.WorkItemExecution dotnet/runtime#122090
1217080 dotnet/runtime System.Runtime.Numerics.Tests.WorkItemExecution
1216955 dotnet/runtime WasmTestOnChrome-ST-Microsoft.Extensions.Configuration.Ini.Tests.WorkItemExecution dotnet/runtime#121906
1216938 dotnet/runtime JIT.CodeGenBringUpTests_d.WorkItemExecution dotnet/runtime#120688
1216897 dotnet/runtime ComInterfaceGenerator.Tests.WorkItemExecution dotnet/runtime#121973
1216753 dotnet/runtime ComInterfaceGenerator.Tests.WorkItemExecution dotnet/runtime#121862
1216342 dotnet/runtime System.Text.Json.SourceGeneration.Roslyn3.11.Tests.WorkItemExecution dotnet/runtime#122070
1216237 dotnet/runtime Microsoft.Extensions.Configuration.Tests.WorkItemExecution dotnet/runtime#121956
1216137 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1216157 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#122068
1216133 dotnet/runtime System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http3.Public_Interop_ExactVersion_BufferContent_Success
1216040 dotnet/runtime Workloads-ST-Wasm.Build.Tests.InterpPgoTests.WorkItemExecution
1216020 dotnet/runtime NoWebcil-ST-Wasm.Build.Tests.WasmTemplateTests.WorkItemExecution dotnet/runtime#122056
1215929 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Intrinsics.Tests.WorkItemExecution dotnet/runtime#120893
1215822 dotnet/runtime NoWebcil-ST-Wasm.Build.Tests.IcuShardingTests.WorkItemExecution dotnet/runtime#122056
1215765 dotnet/runtime ComInterfaceGenerator.Unit.Tests.WorkItemExecution dotnet/runtime#122023
1215705 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Intrinsics.Tests.WorkItemExecution dotnet/runtime#122059
1215665 dotnet/runtime System.Runtime.Numerics.Tests.WorkItemExecution
1215643 dotnet/runtime Common.Tests.WorkItemExecution
1215581 dotnet/runtime Wasm.Build.Tests.SatelliteAssembliesTests.ResourcesFromMainAssembly dotnet/runtime#122056
1215576 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#122054
1215508 dotnet/runtime NoWebcil-ST-Wasm.Build.Tests.WasmTemplateTests.WorkItemExecution
1215470 dotnet/runtime ComInterfaceGenerator.Unit.Tests.WorkItemExecution dotnet/runtime#122023
1215379 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#122053
1215373 dotnet/runtime System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http3.Public_Interop_ExactVersion_BufferContent_Success
1215334 dotnet/runtime Common.Tests.WorkItemExecution
1215368 dotnet/runtime System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http3.Public_Interop_ExactVersion_BufferContent_Success
1215318 dotnet/runtime WasmTestOnChrome-ST-Microsoft.Bcl.Memory.Tests.WorkItemExecution dotnet/runtime#121906
1215222 dotnet/runtime NoWorkload-ST-Wasm.Build.Tests.WorkItemExecution dotnet/runtime#122051
1215187 dotnet/runtime System.Xml.Schema.Extensions.Tests.WorkItemExecution
1215158 dotnet/runtime ComInterfaceGenerator.Unit.Tests.WorkItemExecution dotnet/runtime#122017
1214938 dotnet/runtime NoWebcil-ST-Wasm.Build.Tests.BuildPublishTests.WorkItemExecution dotnet/runtime#121956
1214891 dotnet/runtime Workloads-ST-Wasm.Build.Tests.Blazor.AppsettingsTests.WorkItemExecution dotnet/runtime#120330
1214886 dotnet/runtime System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http3.Public_Interop_ExactVersion_BufferContent_Success
1214765 dotnet/runtime System.Transactions.Local.Tests.WorkItemExecution
1214644 dotnet/runtime IcuAppLocal.Tests.WorkItemExecution dotnet/runtime#120688
1214439 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Intrinsics.Tests.WorkItemExecution dotnet/runtime#116574
1214687 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#121614
1214477 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.InteropServices.JavaScript.Tests.WorkItemExecution dotnet/runtime#122024
1214280 dotnet/runtime NoWebcil-ST-Wasm.Build.NativeRebuild.Tests.FlagsChangeRebuildTests.WorkItemExecution dotnet/runtime#120330
Displaying 100 of 229 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
8 62 229

jozkee avatar Jun 30 '25 17:06 jozkee

Tagging subscribers to 'arch-wasm': @lewing See info in area-owners.md if you want to be subscribed.

Tagging subscribers to this area: @dotnet/runtime-infrastructure See info in area-owners.md if you want to be subscribed.

This is from a preview6 run from Jun 20th, it might've been fixed in main already.

jozkee avatar Jun 30 '25 20:06 jozkee

@akoeplinger do we know what is causing these in general?

lewing avatar Jul 01 '25 17:07 lewing

hm no, this looks like an infrastructure/dnceng issue to me

akoeplinger avatar Jul 03 '25 14:07 akoeplinger

We are hitting a similar one: Unable to pull image mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-helix-sqlserver-amd64 in efcore https://dev.azure.com/dnceng-public/public/_build/results?buildId=1106421&view=logs&j=f95328f7-07f5-5106-5c2a-5dc7069f305c&t=e2f3026d-346a-509b-3549-57236476a9cb https://dev.azure.com/dnceng-public/public/_build/results?buildId=1108065&view=logs&j=f95328f7-07f5-5106-5c2a-5dc7069f305c&t=e2f3026d-346a-509b-3549-57236476a9cb

AndriySvyryd avatar Jul 29 '25 01:07 AndriySvyryd

cc @ericstj

lewing avatar Jul 30 '25 19:07 lewing

If you look at the run_client.py log for the workitem you see this:

2025-07-30T19:06:18.967Z	INFO   	executor(114)	run	Successfully downloaded work item payload
2025-07-30T19:06:18.967Z	INFO   	interval(67)	set_abort_after_timer	Set the timer's abort_after to 5460 seconds from now: abort_after = 2025-07-30 20:37:18.967591
2025-07-30T19:06:19.045Z	INFO   	dockerhelper(70)	ensure_docker_running	Ensuring docker is running: Current status=0
2025-07-30T19:06:19.045Z	INFO   	dockerhelper(76)	ensure_docker_running	Done.
2025-07-30T19:06:19.046Z	INFO   	dockerhelper(22)	cleanup	Cleaning up Docker on the machine
2025-07-30T19:06:19.049Z	INFO   	dockerhelper(39)	pull	Pulling image dotnet-buildtools/prereqs:ubuntu-22.04-helix-webassembly from registry mcr.microsoft.com...
2025-07-30T19:06:20.457Z	WARNING	dockerhelper(47)	pull	Hit exception trying to pull: 404 Client Error: Not Found ("no such image: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-helix-webassembly: No such image: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-helix-webassembly") Retrying up to 5 times
2025-07-30T19:06:30.663Z	WARNING	dockerhelper(47)	pull	Hit exception trying to pull: 404 Client Error: Not Found ("no such image: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-helix-webassembly: No such image: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-helix-webassembly") Retrying up to 5 times
2025-07-30T19:06:40.876Z	WARNING	dockerhelper(47)	pull	Hit exception trying to pull: 404 Client Error: Not Found ("no such image: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-helix-webassembly: No such image: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-helix-webassembly") Retrying up to 5 times
2025-07-30T19:06:51.129Z	WARNING	dockerhelper(47)	pull	Hit exception trying to pull: 404 Client Error: Not Found ("no such image: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-helix-webassembly: No such image: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-helix-webassembly") Retrying up to 5 times
2025-07-30T19:07:02.076Z	WARNING	dockerhelper(47)	pull	Hit exception trying to pull: 404 Client Error: Not Found ("no such image: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-helix-webassembly: No such image: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-helix-webassembly") Retrying up to 5 times
2025-07-30T19:07:12.086Z	ERROR  	dockerhelper(54)	pull	Pull unsuccessful after 5 tries!
2025-07-30T19:07:12.087Z	INFO   	interval(70)	cancel_abort_after	Cancel abort_after for timer
2025-07-30T19:07:12.087Z	INFO   	executor(729)	_execute_command_in_container	The return code was not 0. Sleeping for 30 seconds in case it's still writing to disk.

So for some reason it's getting 404 but this doesn't make much sense.

I see the same thing for the mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-helix-sqlserver-amd64 image that @AndriySvyryd mentioned.

akoeplinger avatar Jul 31 '25 12:07 akoeplinger

@mthalman do you have any idea what would cause these docker pull errors? The images seem to still be present in https://github.com/dotnet/dotnet-buildtools-prereqs-docker. I would expect the tags to always exist.

ericstj avatar Jul 31 '25 19:07 ericstj

Since the tag is correct and the image does exist, it would seem this is a service-level issue. You can write up an issue at https://github.com/microsoft/containerregistry.

mthalman avatar Jul 31 '25 19:07 mthalman

This hits many different images. For example, I just saw Unable to pull image mcr.microsoft.com/dotnet-buildtools/prereqs:debian-13-helix-amd64. I have generalized the error message to include all prereq image. Also, I have set retry to true.

jkotas avatar Aug 01 '25 01:08 jkotas