[7.0] tvos arm64 mono: Failed to list devices

[carlossanlop]

Error Blob

{
  "ErrorMessage": "System.Exception: Failed to list devices",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Reproduction Steps

• PR: https://github.com/dotnet/runtime/pull/84145
• Queue: Build tvOS arm64 Release AllSubsets_Mono
• Job: https://dev.azure.com/dnceng-public/public/_build/results?buildId=228831&view=logs&j=f4520fb1-1559-5885-1d9c-3cb3f6a85e23&t=6c7a8cfe-f92e-569a-eef9-b2ad3e13056d
• Log: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-84145-merge-724493c3e0624d1f85/System.Runtime.Tests/1/console.881c55a3.log?helixlogtype=result
• Output:

 [194/195] System.Private.CoreLib.dll -> System.Private.CoreLib.dll.s, System.Private.CoreLib.dll-llvm.o, System.Private.CoreLib.aotdata
  [195/195] System.Runtime.Tests.dll -> System.Runtime.Tests.dll.s, System.Runtime.Tests.dll-llvm.o, System.Runtime.Tests.aotdata
  Running: cmake -S. -BSystem.Runtime.Tests -GXcode -DCMAKE_SYSTEM_NAME=tvOS -DCMAKE_OSX_DEPLOYMENT_TARGET=10.1
  Using working directory: /tmp/helix/working/A1F0094B/w/96A108E7/e/apple_build/AppBundle/
  Exit code: 0
  Running: xcodebuild ONLY_ACTIVE_ARCH=YES CODE_SIGN_IDENTITY="" CODE_SIGNING_REQUIRED=NO CODE_SIGNING_ALLOWED=NO -arch arm64 -sdk appletvos -configuration Release
  Using working directory: /tmp/helix/working/A1F0094B/w/96A108E7/e/apple_build/AppBundle/System.Runtime.Tests
  Exit code: 0
  
  APP size: 345.9 Mb.
  
  Xcode: /tmp/helix/working/A1F0094B/w/96A108E7/e/apple_build/AppBundle/System.Runtime.Tests/System.Runtime.Tests.xcodeproj
  App: /tmp/helix/working/A1F0094B/w/96A108E7/e/apple_build/AppBundle/System.Runtime.Tests/Release-appletvos/System.Runtime.Tests.app
  App moved to: /private/tmp/helix/working/A1F0094B/w/96A108E7/e/System.Runtime.Tests.app/
Signing /tmp/helix/working/A1F0094B/w/96A108E7/e/System.Runtime.Tests.app
    "/Users/helix-runner/Library/Keychains/signing-certs.keychain-db"
/tmp/helix/working/A1F0094B/w/96A108E7/e/System.Runtime.Tests.app: signed app bundle with Mach-O thin (arm64) [net.dot.System.Runtime.Tests]
[1.0.0-prerelease.22411.1+5ebf69650b9f7b4ecab485be840b3022420f7812] XHarness command issued: apple test --app /tmp/helix/working/A1F0094B/w/96A108E7/e/System.Runtime.Tests.app --output-directory /tmp/helix/working/A1F0094B/w/96A108E7/uploads --target tvos-device --timeout 00:45:00 --xcode /Applications/Xcode_14.1.app -v --launch-timeout 00:05:00 --signal-app-end --
[15:14:06] info: Preparing run for tvos-device
[15:14:06] info: Looking for available tvos-device devices..
[15:14:06] dbug: Looking for available tvos-device devices. Storing logs into list-tvos-device-20230405_151406.log
[15:14:06] crit: System.Exception: Failed to list devices.
                    at Microsoft.DotNet.XHarness.iOS.Shared.Hardware.HardwareDeviceLoader.LoadDevices(ILog log, Boolean includeLocked, Boolean forceRefresh, Boolean listExtraData, Boolean includeWirelessDevices, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.iOS.Shared/Hardware/HardwareDeviceLoader.cs:line 103
                    at Microsoft.DotNet.XHarness.Apple.DeviceFinder.FindDevice(TestTargetOs target, String deviceName, ILog log, Boolean includeWirelessDevices, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/DeviceFinder.cs:line 72
                    at Microsoft.DotNet.XHarness.Apple.BaseOrchestrator.OrchestrateOperationInternal(TestTargetOs target, String deviceName, Boolean includeWirelessDevices, Boolean resetSimulator, Boolean enableLldb, GetAppBundleInfoFunc getAppBundle, ExecuteMacCatalystAppFunc executeMacCatalystApp, ExecuteAppFunc executeApp, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/Orchestration/BaseOrchestrator.cs:line 215
                    at Microsoft.DotNet.XHarness.Apple.BaseOrchestrator.OrchestrateOperation(TestTargetOs target, String deviceName, Boolean includeWirelessDevices, Boolean resetSimulator, Boolean enableLldb, GetAppBundleInfoFunc getAppBundle, ExecuteMacCatalystAppFunc executeMacCatalystApp, ExecuteAppFunc executeApp, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/Orchestration/BaseOrchestrator.cs:line 96
                    at Microsoft.DotNet.XHarness.Apple.TestOrchestrator.OrchestrateTest(GetAppBundleInfoFunc getAppBundlePath, TestTargetOs target, String deviceName, TimeSpan timeout, TimeSpan launchTimeout, CommunicationChannel communicationChannel, XmlResultJargon xmlResultJargon, IEnumerable`1 singleMethodFilters, IEnumerable`1 classMethodFilters, Boolean includeWirelessDevices, Boolean resetSimulator, Boolean enableLldb, Boolean signalAppEnd, IReadOnlyCollection`1 environmentalVariables, IEnumerable`1 passthroughArguments, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/Orchestration/TestOrchestrator.cs:line 189
                    at Microsoft.DotNet.XHarness.CLI.Commands.Apple.AppleAppCommand`1.Invoke(ILogger logger) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/Apple/AppleAppCommand.cs:line 68
                    at Microsoft.DotNet.XHarness.CLI.Commands.XHarnessCommand`1.Invoke(IEnumerable`1 arguments) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/XHarnessCommand.cs:line 145
XHarness exit code: 71 (GENERAL_FAILURE)

Report

Build	Definition	Test	Pull Request
528334	dotnet/runtime	System.Console.Tests.WorkItemExecution
524193	dotnet/runtime	tvOS.Device.Aot.Test.WorkItemExecution	dotnet/runtime#96858
513152	dotnet/runtime	System.IO.FileSystem.DisabledFileLocking.Tests.WorkItemExecution
509716	dotnet/runtime	Microsoft.Extensions.Diagnostics.Tests.WorkItemExecution
507446	dotnet/runtime	System.Resources.Reader.Tests.WorkItemExecution	dotnet/runtime#96270

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	1	5

Known issue validation

Build: :mag_right: Result validation: :warning: Validation could not be done without an Azure DevOps build URL on the issue. Please add it to the "Build: :mag_right:" line.

[ghost]

Tagging subscribers to 'os-tvos': @steveisok, @akoeplinger See info in area-owners.md if you want to be subscribed.

Issue Details

---

Error Blob

{
  "ErrorMessage": "crit: System.Exception: Failed to list devices.",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Reproduction Steps

• PR: https://github.com/dotnet/runtime/pull/84145
• Queue: Build tvOS arm64 Release AllSubsets_Mono
• Job: https://dev.azure.com/dnceng-public/public/_build/results?buildId=228831&view=logs&j=f4520fb1-1559-5885-1d9c-3cb3f6a85e23&t=6c7a8cfe-f92e-569a-eef9-b2ad3e13056d
• Log: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-84145-merge-724493c3e0624d1f85/System.Runtime.Tests/1/console.881c55a3.log?helixlogtype=result
• Output:

 [194/195] System.Private.CoreLib.dll -> System.Private.CoreLib.dll.s, System.Private.CoreLib.dll-llvm.o, System.Private.CoreLib.aotdata
  [195/195] System.Runtime.Tests.dll -> System.Runtime.Tests.dll.s, System.Runtime.Tests.dll-llvm.o, System.Runtime.Tests.aotdata
  Running: cmake -S. -BSystem.Runtime.Tests -GXcode -DCMAKE_SYSTEM_NAME=tvOS -DCMAKE_OSX_DEPLOYMENT_TARGET=10.1
  Using working directory: /tmp/helix/working/A1F0094B/w/96A108E7/e/apple_build/AppBundle/
  Exit code: 0
  Running: xcodebuild ONLY_ACTIVE_ARCH=YES CODE_SIGN_IDENTITY="" CODE_SIGNING_REQUIRED=NO CODE_SIGNING_ALLOWED=NO -arch arm64 -sdk appletvos -configuration Release
  Using working directory: /tmp/helix/working/A1F0094B/w/96A108E7/e/apple_build/AppBundle/System.Runtime.Tests
  Exit code: 0
  
  APP size: 345.9 Mb.
  
  Xcode: /tmp/helix/working/A1F0094B/w/96A108E7/e/apple_build/AppBundle/System.Runtime.Tests/System.Runtime.Tests.xcodeproj
  App: /tmp/helix/working/A1F0094B/w/96A108E7/e/apple_build/AppBundle/System.Runtime.Tests/Release-appletvos/System.Runtime.Tests.app
  App moved to: /private/tmp/helix/working/A1F0094B/w/96A108E7/e/System.Runtime.Tests.app/
Signing /tmp/helix/working/A1F0094B/w/96A108E7/e/System.Runtime.Tests.app
    "/Users/helix-runner/Library/Keychains/signing-certs.keychain-db"
/tmp/helix/working/A1F0094B/w/96A108E7/e/System.Runtime.Tests.app: signed app bundle with Mach-O thin (arm64) [net.dot.System.Runtime.Tests]
[1.0.0-prerelease.22411.1+5ebf69650b9f7b4ecab485be840b3022420f7812] XHarness command issued: apple test --app /tmp/helix/working/A1F0094B/w/96A108E7/e/System.Runtime.Tests.app --output-directory /tmp/helix/working/A1F0094B/w/96A108E7/uploads --target tvos-device --timeout 00:45:00 --xcode /Applications/Xcode_14.1.app -v --launch-timeout 00:05:00 --signal-app-end --
[15:14:06] info: Preparing run for tvos-device
[15:14:06] info: Looking for available tvos-device devices..
[15:14:06] dbug: Looking for available tvos-device devices. Storing logs into list-tvos-device-20230405_151406.log
[15:14:06] crit: System.Exception: Failed to list devices.
                    at Microsoft.DotNet.XHarness.iOS.Shared.Hardware.HardwareDeviceLoader.LoadDevices(ILog log, Boolean includeLocked, Boolean forceRefresh, Boolean listExtraData, Boolean includeWirelessDevices, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.iOS.Shared/Hardware/HardwareDeviceLoader.cs:line 103
                    at Microsoft.DotNet.XHarness.Apple.DeviceFinder.FindDevice(TestTargetOs target, String deviceName, ILog log, Boolean includeWirelessDevices, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/DeviceFinder.cs:line 72
                    at Microsoft.DotNet.XHarness.Apple.BaseOrchestrator.OrchestrateOperationInternal(TestTargetOs target, String deviceName, Boolean includeWirelessDevices, Boolean resetSimulator, Boolean enableLldb, GetAppBundleInfoFunc getAppBundle, ExecuteMacCatalystAppFunc executeMacCatalystApp, ExecuteAppFunc executeApp, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/Orchestration/BaseOrchestrator.cs:line 215
                    at Microsoft.DotNet.XHarness.Apple.BaseOrchestrator.OrchestrateOperation(TestTargetOs target, String deviceName, Boolean includeWirelessDevices, Boolean resetSimulator, Boolean enableLldb, GetAppBundleInfoFunc getAppBundle, ExecuteMacCatalystAppFunc executeMacCatalystApp, ExecuteAppFunc executeApp, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/Orchestration/BaseOrchestrator.cs:line 96
                    at Microsoft.DotNet.XHarness.Apple.TestOrchestrator.OrchestrateTest(GetAppBundleInfoFunc getAppBundlePath, TestTargetOs target, String deviceName, TimeSpan timeout, TimeSpan launchTimeout, CommunicationChannel communicationChannel, XmlResultJargon xmlResultJargon, IEnumerable`1 singleMethodFilters, IEnumerable`1 classMethodFilters, Boolean includeWirelessDevices, Boolean resetSimulator, Boolean enableLldb, Boolean signalAppEnd, IReadOnlyCollection`1 environmentalVariables, IEnumerable`1 passthroughArguments, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/Orchestration/TestOrchestrator.cs:line 189
                    at Microsoft.DotNet.XHarness.CLI.Commands.Apple.AppleAppCommand`1.Invoke(ILogger logger) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/Apple/AppleAppCommand.cs:line 68
                    at Microsoft.DotNet.XHarness.CLI.Commands.XHarnessCommand`1.Invoke(IEnumerable`1 arguments) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/XHarnessCommand.cs:line 145
XHarness exit code: 71 (GENERAL_FAILURE)

Author:	carlossanlop
Assignees:	-
Labels:	arch-arm64, blocking-clean-ci, area-VM-meta-mono, runtime-mono, os-tvos, Known Build Error
Milestone:	7.0.x

[carlossanlop]

Not sure why it's saying there are zero hits, I'm still seeing this failure. Example:

• Affected 7.0 PR: https://github.com/dotnet/runtime/pull/84648
• Queue: Build tvOS arm64 Release AllSubsets_Mono
• Job: https://dev.azure.com/dnceng-public/public/_build/results?buildId=235364&view=logs&j=f4520fb1-1559-5885-1d9c-3cb3f6a85e23&t=6c7a8cfe-f92e-569a-eef9-b2ad3e13056d&l=57
• Log file: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-84648-merge-9ee17d6a2fcc4e929f/System.Runtime.Tests/1/console.8e130f2a.log?helixlogtype=result

[20:59:02] info: Preparing run for tvos-device
[20:59:02] info: Looking for available tvos-device devices..
[20:59:03] dbug: Looking for available tvos-device devices. Storing logs into list-tvos-device-20230411_205902.log
[20:59:03] crit: System.Exception: Failed to list devices.
                    at Microsoft.DotNet.XHarness.iOS.Shared.Hardware.HardwareDeviceLoader.LoadDevices(ILog log, Boolean includeLocked, Boolean forceRefresh, Boolean listExtraData, Boolean includeWirelessDevices, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.iOS.Shared/Hardware/HardwareDeviceLoader.cs:line 103
                    at Microsoft.DotNet.XHarness.Apple.DeviceFinder.FindDevice(TestTargetOs target, String deviceName, ILog log, Boolean includeWirelessDevices, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/DeviceFinder.cs:line 72
                    at Microsoft.DotNet.XHarness.Apple.BaseOrchestrator.OrchestrateOperationInternal(TestTargetOs target, String deviceName, Boolean includeWirelessDevices, Boolean resetSimulator, Boolean enableLldb, GetAppBundleInfoFunc getAppBundle, ExecuteMacCatalystAppFunc executeMacCatalystApp, ExecuteAppFunc executeApp, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/Orchestration/BaseOrchestrator.cs:line 215
                    at Microsoft.DotNet.XHarness.Apple.BaseOrchestrator.OrchestrateOperation(TestTargetOs target, String deviceName, Boolean includeWirelessDevices, Boolean resetSimulator, Boolean enableLldb, GetAppBundleInfoFunc getAppBundle, ExecuteMacCatalystAppFunc executeMacCatalystApp, ExecuteAppFunc executeApp, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/Orchestration/BaseOrchestrator.cs:line 96
                    at Microsoft.DotNet.XHarness.Apple.TestOrchestrator.OrchestrateTest(GetAppBundleInfoFunc getAppBundlePath, TestTargetOs target, String deviceName, TimeSpan timeout, TimeSpan launchTimeout, CommunicationChannel communicationChannel, XmlResultJargon xmlResultJargon, IEnumerable`1 singleMethodFilters, IEnumerable`1 classMethodFilters, Boolean includeWirelessDevices, Boolean resetSimulator, Boolean enableLldb, Boolean signalAppEnd, IReadOnlyCollection`1 environmentalVariables, IEnumerable`1 passthroughArguments, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/Orchestration/TestOrchestrator.cs:line 189
                    at Microsoft.DotNet.XHarness.CLI.Commands.Apple.AppleAppCommand`1.Invoke(ILogger logger) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/Apple/AppleAppCommand.cs:line 68
                    at Microsoft.DotNet.XHarness.CLI.Commands.XHarnessCommand`1.Invoke(IEnumerable`1 arguments) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/XHarnessCommand.cs:line 145
XHarness exit code: 71 (GENERAL_FAILURE)

[carlossanlop]

Also hit in this other 7.0 PR: https://github.com/dotnet/runtime/pull/84658 Job: https://dev.azure.com/dnceng-public/public/_build/results?buildId=235361&view=logs&j=f4520fb1-1559-5885-1d9c-3cb3f6a85e23&t=6c7a8cfe-f92e-569a-eef9-b2ad3e13056d

[steveisok]

I think we'll need to bump xcode on these machines.

[jeffschwMSFT]

removing blocking-clean-ci as it has not failed in 30 days

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

[vitek-karas]

Closing as 7.0 only - if we hit it in some other branch, let's create a new issue for it.