sdk icon indicating copy to clipboard operation
sdk copied to clipboard

Published 20 hours ago • verified publisher icon dart-lang

Flaky VM crash: runtime\platform\threads.h: 55: error: expected: owner_ == kUnowned

Open alexmarkov opened this issue 8 months ago • 6 comments

The following flaky crash happened on vm-aot-win-debug-arm64:

/===========================================================================\
| service/dev_fs_http_put_test/service broke (Pass -> Crash, expected Pass) |
\===========================================================================/

--- Command "vm_compile_to_kernel" (took 05.000785s):
set DART_CONFIGURATION=DebugARM64 & set DART_SUPPRESS_WER=1 & set DART_CRASHPAD_HANDLER=c:\opt\s\w\ir\cache\builder\sdk\out\DebugARM64\crashpad_handler.exe & C:\opt\s\w\ir\cache\builder\sdk\pkg\vm\tool\gen_kernel.bat --aot --platform=out/DebugARM64/vm_platform_strong.dill -o c:\opt\s\w\ir\cache\builder\sdk\out\DebugARM64\generated_compilations\vm-aot-win-debug-arm64\runtime_observatory_tests_service_dev_fs_http_put_test\out.dill C:\opt\s\w\ir\cache\builder\sdk\runtime\observatory\tests\service\dev_fs_http_put_test.dart -Dtest_runner.configuration=vm-aot-win-debug-arm64 --packages=C:\opt\s\w\ir\cache\builder\sdk\.dart_tool\package_config.json -Ddart.vm.product=false --sound-null-safety

exit code:
0

--- Command "precompiler" (took 23.000218s):
set DART_CONFIGURATION=DebugARM64 & set DART_SUPPRESS_WER=1 & set DART_CRASHPAD_HANDLER=c:\opt\s\w\ir\cache\builder\sdk\out\DebugARM64\crashpad_handler.exe & out\DebugARM64\gen_snapshot --snapshot-kind=app-aot-elf --elf=c:/opt/s/w/ir/cache/builder/sdk/out/DebugARM64/generated_compilations/vm-aot-win-debug-arm64/runtime_observatory_tests_service_dev_fs_http_put_test/out.aotsnapshot --loading-unit-manifest=c:/opt/s/w/ir/cache/builder/sdk/out/DebugARM64/generated_compilations/vm-aot-win-debug-arm64/runtime_observatory_tests_service_dev_fs_http_put_test/ignored.json --sound-null-safety -Dtest_runner.configuration=vm-aot-win-debug-arm64 --ignore-unrecognized-flags --packages=C:\opt\s\w\ir\cache\builder\sdk\.dart_tool\package_config.json c:\opt\s\w\ir\cache\builder\sdk\out\DebugARM64\generated_compilations\vm-aot-win-debug-arm64\runtime_observatory_tests_service_dev_fs_http_put_test\out.dill

exit code:
0

--- Command "remove_kernel_file" (took 44ms):
set DART_CONFIGURATION=DebugARM64 & set DART_SUPPRESS_WER=1 & set DART_CRASHPAD_HANDLER=c:\opt\s\w\ir\cache\builder\sdk\out\DebugARM64\crashpad_handler.exe & cmd.exe /c del c:\opt\s\w\ir\cache\builder\sdk\out\DebugARM64\generated_compilations\vm-aot-win-debug-arm64\runtime_observatory_tests_service_dev_fs_http_put_test\out.dill

exit code:
0

--- Command "vm" (took 296ms):
set DART_CONFIGURATION=DebugARM64 & set DART_SUPPRESS_WER=1 & set DART_CRASHPAD_HANDLER=c:\opt\s\w\ir\cache\builder\sdk\out\DebugARM64\crashpad_handler.exe & set TEST_COMPILATION_DIR=c:/opt/s/w/ir/cache/builder/sdk/out/DebugARM64/generated_compilations/vm-aot-win-debug-arm64/runtime_observatory_tests_service_dev_fs_http_put_test & out\DebugARM64\dartaotruntime.exe --sound-null-safety -Dtest_runner.configuration=vm-aot-win-debug-arm64 --ignore-unrecognized-flags --packages=C:\opt\s\w\ir\cache\builder\sdk\.dart_tool\package_config.json c:/opt/s/w/ir/cache/builder/sdk/out/DebugARM64/generated_compilations/vm-aot-win-debug-arm64/runtime_observatory_tests_service_dev_fs_http_put_test/out.aotsnapshot

exit code:
3

stdout:
00:00 [32m+0[0m: out.aotsnapshot (VM Service)[0m

** Launching SERVICE_TEST_TESTEE=true out\DebugARM64\dartaotruntime.exe --disable-dart-dev --no-dds --write-service-info=file:///c:/opt/s/w/ir/x/t/dart_service40eeedfd/service_info.json --profiler --sound-null-safety -Dtest_runner.configuration=vm-aot-win-debug-arm64 --ignore-unrecognized-flags --packages=C:\opt\s\w\ir\cache\builder\sdk\.dart_tool\package_config.json --enable-vm-service:0 c:\opt\s\w\ir\cache\builder\sdk\out\DebugARM64\generated_compilations\vm-aot-win-debug-arm64\runtime_observatory_tests_service_dev_fs_http_put_test\out.aotsnapshot

>testee>out> 

>testee>out> The Dart VM service is listening on http://127.0.0.1:50716/RpAsqRZCj0E=/

** Signaled to run test queries on http://127.0.0.1:50716/RpAsqRZCj0E= (VM Service)

Loading VM...

Done loading VM

Running out.aotsnapshot [1/1]

All service tests completed successfully.

** Killing script

00:00 [32m+1[0m: All tests passed![0m

** Process exited

stderr:
c:\opt\s\w\ir\cache\builder\sdk\runtime\platform\threads.h: 55: error: expected: owner_ == kUnowned
version=3.8.0-edge.19e2a2d58c9574fbf4a3a9658eb76072c0461833 (main) (Mon Apr 7 13:48:53 2025 -0700) on "windows_arm64"
pid=11864, thread=3056, isolate_group=(nil)(0000000000000000), isolate=(nil)(0000000000000000)
os=windows, arch=arm64, comp=no, sim=no
isolate_instructions=0, vm_instructions=0
fp=8c0b6ff7d0, sp=8c0b6ff7c0, pc=7ff6d8caceec
  pc 0x00007ff6d8caceec fp 0x0000008c0b6ff7d0 Unknown symbol
  pc 0x00007ff6d8093ce8 fp 0x0000008c0b6ff7f0 Unknown symbol
  pc 0x00007ff6d8095148 fp 0x0000008c0b6ff830 Unknown symbol
  pc 0x00007ff6d80be49c fp 0x0000008c0b6ff850 Unknown symbol
  pc 0x00007ff6d80824d0 fp 0x0000008c0b6ff880 Unknown symbol
  pc 0x00007ff6d8093468 fp 0x0000008c0b6ff8a0 Unknown symbol
  pc 0x00007ff6d808f0ac fp 0x0000008c0b6ffad0 Unknown symbol
  pc 0x00007ff6d8fb9be0 fp 0x0000008c0b6ffaf0 Unknown symbol
  pc 0x00007ff6d8fb98c4 fp 0x0000008c0b6ffb20 Unknown symbol
  pc 0x00007ff6d8fb96e4 fp 0x0000008c0b6ffb60 Unknown symbol
  pc 0x00007ff6d8fb9dac fp 0x0000008c0b6ffb70 Unknown symbol
  pc 0x00007ff848762310 fp 0x0000008c0b6ffb90 Unknown symbol
  pc 0x00007ff849055aec fp 0x0000008c0b6ffbd0 Unknown symbol
-- End of DumpStackTrace

--- Re-run this test:
python3 tools/test.py -n vm-aot-win-debug-arm64 service/dev_fs_http_put_test/service

https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket/8718184469624617057/+/u/test_results/ignored_flaky_test_failure_logs

/cc @rmacnak-google

alexmarkov avatar Apr 08 '25 15:04 alexmarkov

Similar crash happened on vm-win-debug-x64 about a week ago. So it is extremely rare and not specific to a particular test.

/===========================================================================\
| co19/LibTest/io/Stdout/write_A01_t07 broke (Pass -> Crash, expected Pass) |
\===========================================================================/

--- Command "vm" (took 02.000787s):
set DART_CONFIGURATION=DebugX64 & set DART_SUPPRESS_WER=1 & set DART_CRASHPAD_HANDLER=C:\b\s\w\ir\out\DebugX64\crashpad_handler.exe & out\DebugX64\dart.exe --sound-null-safety -Dtest_runner.configuration=vm-win-debug-x64 --ignore-unrecognized-flags --packages=C:\b\s\w\ir\.dart_tool\package_config.json C:\b\s\w\ir\tests\co19\src\LibTest\io\Stdout\write_A01_t07.dart

exit code:
1073741845

stderr:
../../runtime/platform/threads.h: 55: error: expected: owner_ == kUnowned
version=3.8.0-edge (main) (Unknown timestamp) on "windows_x64"
pid=3360, thread=4808, isolate_group=(nil)(0000000000000000), isolate=(nil)(0000000000000000)
os=windows, arch=x64, comp=no, sim=no
isolate_instructions=0, vm_instructions=0
fp=a77e72f828, sp=a77e72f828, pc=7ff61a7c91c2
  pc 0x00007ff61a7c91c2 fp 0x000000a77e72f828 Unknown symbol
-- End of DumpStackTrace

--- Re-run this test:
python3 tools/test.py -n vm-win-debug-x64 co19/LibTest/io/Stdout/write_A01_t07

https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket/8718773968426090561/+/u/test_results/ignored_flaky_test_failure_logs

alexmarkov avatar Apr 08 '25 15:04 alexmarkov

One more crash happened on March 20, on vm-win-debug-x64 bot and different test:

/================================================================================\
| pkg/vm_service/test/eval_issue_49209_test broke (Pass -> Crash, expected Pass) |
\================================================================================/

--- Command "vm" (took 13.000138s):
set DART_CONFIGURATION=DebugX64 & set DART_SUPPRESS_WER=1 & set DART_CRASHPAD_HANDLER=C:\b\s\w\ir\out\DebugX64\crashpad_handler.exe & out\DebugX64\dart.exe --sound-null-safety -Dtest_runner.configuration=vm-win-debug-x64 --ignore-unrecognized-flags --packages=C:\b\s\w\ir\.dart_tool\package_config.json C:\b\s\w\ir\pkg\vm_service\test\eval_issue_49209_test.dart

exit code:
1073741845

stdout:
00:00 [32m+0[0m: eval_issue_49209_test.dart[0m

** Launching SERVICE_TEST_TESTEE=true out\DebugX64\dart.exe --disable-service-auth-codes --profiler --sound-null-safety -Dtest_runner.configuration=vm-win-debug-x64 --ignore-unrecognized-flags --packages=C:\b\s\w\ir\.dart_tool\package_config.json --enable-vm-service:0 C:\b\s\w\ir\pkg\vm_service\test\eval_issue_49209_test.dart

>testee>out> The Dart VM service is listening on http://127.0.0.1:58433//
>testee>out> The Dart DevTools debugger and profiler is available at: http://127.0.0.1:58433/devtools/?uri=ws://127.0.0.1:58433/ws
** Signaled to run test queries on http://127.0.0.1:58433//

>testee>out> 
>testee>out> A<C>
Done loading VM

eval_issue_49209_test.dart [1/2]

eval_issue_49209_test.dart [2/2]

[InstanceRef id: objects/4/0, kind: String, identityHashCode: 0, classRef: [ClassRef id: classes/94, name: _OneByteString, library: [LibraryRef id: libraries/@0150898, name: dart.core, uri: dart:core]]]

All service tests completed successfully.

** Killing script

00:06 [32m+1[0m: All tests passed![0m

** Process exited: -1

stderr:
../../runtime/platform/threads.h: 55: error: expected: owner_ == kUnowned
version=3.8.0-edge (main) (Unknown timestamp) on "windows_x64"
pid=4320, thread=7332, isolate_group=(nil)(0000000000000000), isolate=(nil)(0000000000000000)
os=windows, arch=x64, comp=no, sim=no
isolate_instructions=0, vm_instructions=0
fp=1647d6f918, sp=1647d6f918, pc=7ff7aaba8702
  pc 0x00007ff7aaba8702 fp 0x0000001647d6f918 Unknown symbol
-- End of DumpStackTrace

--- Re-run this test:
python3 tools/test.py -n vm-win-debug-x64 pkg/vm_service/test/eval_issue_49209_test

https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket/8719872653958537345/+/u/test_results/ignored_flaky_test_failure_logs

alexmarkov avatar Apr 08 '25 15:04 alexmarkov

@rmacnak-google has added additional information to be printed on IA32 or arm64 architectures, we will see if this crash reproduces on one of these architectures to be able to debug further.

a-siva avatar Apr 23 '25 20:04 a-siva

Failed on windows_x64 once again:

../../runtime/platform/threads.h: 55: error: expected: owner_ == kUnowned
version=3.9.0-edge (main) (Unknown timestamp) on "windows_x64"
pid=10012, thread=8376, isolate_group=(nil)(0000000000000000), isolate=(nil)(0000000000000000)
os=windows, arch=x64, comp=no, sim=no
isolate_instructions=0, vm_instructions=0
fp=396b51f5c0, sp=396b51f598, pc=7ff683999ccc
  pc 0x00007ff683999ccc fp 0x000000396b51f5c0 Unknown symbol
-- End of DumpStackTrace

--- Re-run this test:
python3 tools/test.py -n vm-win-debug-x64 pkg/vm_service/test/evaluate_optimized_out_variable_test

https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket/8716244609293816657/+/u/test_results/ignored_flaky_test_failure_logs

alexmarkov avatar Apr 30 '25 15:04 alexmarkov

Another failure:

/==============================================================================\
| co19/LibTest/io/Stdout/writeAll_A02_t05 broke (Pass -> Crash, expected Pass) |
\==============================================================================/

--- Command "vm" (took 03.000117s):
set DART_CONFIGURATION=DebugX64 & set DART_SUPPRESS_WER=1 & set DART_CRASHPAD_HANDLER=C:\b\s\w\ir\out\DebugX64\crashpad_handler.exe & out\DebugX64\dart.exe --sound-null-safety -Dtest_runner.configuration=vm-win-debug-x64 --ignore-unrecognized-flags --packages=C:\b\s\w\ir\.dart_tool\package_config.json C:\b\s\w\ir\tests\co19\src\LibTest\io\Stdout\writeAll_A02_t05.dart

exit code:
1073741845

stderr:
../../runtime/platform/threads.h: 55: error: expected: owner_ == kUnowned
version=3.9.0-edge (main) (Unknown timestamp) on "windows_x64"
pid=4380, thread=7572, isolate_group=(nil)(0000000000000000), isolate=(nil)(0000000000000000)
os=windows, arch=x64, comp=no, sim=no
isolate_instructions=0, vm_instructions=0
fp=a1538ff670, sp=a1538ff648, pc=7ff74029dd6c
  pc 0x00007ff74029dd6c fp 0x000000a1538ff670 Unknown symbol
-- End of DumpStackTrace

--- Re-run this test:
python3 tools/test.py -n vm-win-debug-x64 co19/LibTest/io/Stdout/writeAll_A02_t05

https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket/8714231470474498609/+/u/test_results/ignored_flaky_test_failure_logs

alexmarkov avatar May 22 '25 14:05 alexmarkov 

stderr:
../../runtime/platform/threads.h: 55: error: expected: owner_ == kUnowned
version=3.9.0-edge (main) (Unknown timestamp) on "windows_x64"
pid=5944, thread=3948, isolate_group=(nil)(0000000000000000), isolate=(nil)(0000000000000000)
os=windows, arch=x64, comp=no, sim=no
isolate_instructions=0, vm_instructions=0
fp=a21a2ff650, sp=a21a2ff628, pc=7ff6d9e3b99c
  pc 0x00007ff6d9e3b99c fp 0x000000a21a2ff650 Unknown symbol
-- End of DumpStackTrace

--- Re-run this test:
python3 tools/test.py -n vm-aot-win-debug-x64 vm/dart/date_pre1970_test

https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket/8713111031565572977/+/u/test_results/ignored_flaky_test_failure_logs

alexmarkov avatar Jun 09 '25 14:06 alexmarkov

Happened once again on vm-aot-win-debug-x64 bot:

stderr:
../../runtime/platform/threads.h: 55: error: expected: owner_ == kUnowned
version=3.11.0-edge (main) (Unknown timestamp) on "windows_x64"
pid=1816, thread=6468, isolate_group=(nil)(0000000000000000), isolate=(nil)(0000000000000000)
os=windows, arch=x64, comp=no, sim=no
isolate_instructions=0, vm_instructions=0
fp=7ed036f920, sp=7ed036f8d8, pc=7ff77786a523
  pc 0x00007ff77786a523 fp 0x0000007ed036f920 Unknown symbol
-- End of DumpStackTrace

--- Re-run this test:
python3 tools/test.py -n vm-aot-win-debug-x64 pkg/vm_service/test/regexp_function_test/0

log

alexmarkov avatar Oct 23 '25 16:10 alexmarkov 

stderr:
../../runtime/platform/threads.h: 55: error: expected: owner_ == kUnowned
version=3.11.0-edge (main) (Unknown timestamp) on "windows_arm64"
pid=4844, thread=21160, isolate_group=(nil)(0000000000000000), isolate=(nil)(0000000000000000)
os=windows, arch=arm64, comp=no, sim=no
isolate_instructions=0, vm_instructions=0
fp=eeb877f650, sp=eeb877f628, pc=7ff63a76822c
  pc 0x00007ff63a76822c fp 0x000000eeb877f650 Unknown symbol
  pc 0x00007ff63aa3de2c fp 0x000000eeb877f670 Unknown symbol
  pc 0x00007ff63a08a60c fp 0x000000eeb877f6d0 Unknown symbol
  pc 0x00007ff63a0a1868 fp 0x000000eeb877f710 Unknown symbol
  pc 0x00007ff63a0b5084 fp 0x000000eeb877f720 Unknown symbol
  pc 0x00007ff63a085250 fp 0x000000eeb877f948 Unknown symbol
  pc 0x00007ff63a0842d8 fp 0x000000eeb877f970 Unknown symbol
  pc 0x00007ff63a928080 fp 0x000000eeb877f980 Unknown symbol
  pc 0x3a057ff63a928224 fp 0x000000eeb877f9b0 Unknown symbol
  pc 0x950e7ff63a9282e4 fp 0x000000eeb877f9f0 Unknown symbol
  pc 0x33177ff63a92830c fp 0x000000eeb877fa00 Unknown symbol
  pc 0x05627fff6dbb8740 fp 0x000000eeb877fa20 Unknown symbol
  pc 0x0a52ffff6e2c4464 fp 0x000000eeb877fa30 Unknown symbol
-- End of DumpStackTrace

--- Re-run this test:
python3 tools/test.py -n vm-aot-win-debug-arm64 vm/dart/sdk_hash_test

log

alexmarkov avatar Nov 20 '25 16:11 alexmarkov