sdk
sdk copied to clipboard
dart-lang
VM crash: runtime/vm/profiler.cc: 141: error: expected: sample_buffer_ != nullptr
From dart-fuzz bot:
Isolate (/b/s/w/itvz6b0bsx/dart_fuzzHIKQPK) NO-FP NO-FFI FLAT : JIT-DebugSIMRISCV64 - JIT-DebugSIMARM64C: !DIVERGENCE! 1.101:1264572494 (-6 vs 0)
fail1:
-6
{VF5Knej: JtP, 1PaTG: Tx}
... skipped ...
var97: Expando:
print() throws
../../runtime/vm/profiler.cc: 141: error: expected: sample_buffer_ != nullptr
===== CRASH =====
si_signo=Segmentation fault(11), si_code=SEGV_MAPERR(1), si_addr=0xababb2eb
Aborting reentrant request for stack trace.
-- BEGIN REPRODUCE --
DART SDK REVISION:
dart runtime/tools/dartfuzz/dartfuzz.dart --no-fp --no-ffi --flat --seed 1264572494 fuzz.dart
-- RUN 1 --
out/DebugSIMRISCV64/dart --profiler --profile_period=641 --runtime_allocate_spill_tlab --force_evacuation --old_gen_heap_size=128 /b/s/w/itvz6b0bsx/dart_fuzzHIKQPK/fuzz.dart
-- RUN 2 --
out/DebugSIMARM64C/dart --profiler --profile_vm=false --sample_buffer_duration=45 --no_concurrent_sweep --no_unopt_megamorphic_calls --optimization_counter_threshold=20690 --old_gen_heap_size=128 /b/s/w/itvz6b0bsx/dart_fuzzHIKQPK/fuzz.dart
-- END REPRODUCE --
https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket/8713657647748859297/+/u/collect_shards/dartfuzz_-_generated_programs_shard_21/task_stdout_stderr:dartfuzz-_generated_programs_shard_21
/cc @bkonyi @rmacnak-google
happened again https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket/8713204709979493633/+/u/collect_shards/dartfuzz_-_generated_programs_shard_16/task_stdout_stderr:dartfuzz-_generated_programs_shard_16
Still failing:
../../runtime/vm/profiler.cc: 141: error: expected: sample_buffer_ != nullptr
===== CRASH =====
si_signo=Segmentation fault(11), si_code=SEGV_MAPERR(1), si_addr=0xababb303
Aborting reentrant request for stack trace.
-- BEGIN REPRODUCE --
DART SDK REVISION:
dart runtime/tools/dartfuzz/dartfuzz.dart --no-fp --no-ffi --flat --seed 1522580635 fuzz.dart
-- RUN 1 --
out/DebugSIMARM64/dart --profiler --force_clone_compiler_objects --old_gen_heap_size=128 /b/s/w/iti8wgtdzg/dart_fuzzAZWEXV/fuzz.dart
-- RUN 2 --
out/DebugX64/dart --profiler --max_profile_depth=49 --dontneed_on_sweep --old_gen_heap_size=128 /b/s/w/iti8wgtdzg/dart_fuzzAZWEXV/fuzz.dart
-- END REPRODUCE --
https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket/8713113813214013105/+/u/collect_shards/dartfuzz_-_generated_programs_shard_32/task_stdout_stderr:dartfuzz-_generated_programs_shard_32
it is still happening https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket/8710667624351081873/+/u/collect_shards/dartfuzz_-_generated_programs_shard_29/task_stdout_stderr:dartfuzz-_generated_programs_shard_29
Found again here https://ci.chromium.org/ui/p/dart/builders/ci.sandbox/fuzz-linux/4647/overview
I previously thought that this crash was related to CPU sample streaming, but Profiler::ProcessCompletedBlocks returns immediately there is nothing listening to the Profiler stream, and in the generated fuzz.dart programs, nothing listens to the Profiler stream. I have not been able to reproduce this crash locally. My best guess right now is that FlushSampleBlocks is involved, because it sets both isolate->current_sample_block_ and isolate->current_allocation_sample_block_ to nullptr.
I consider this to be related to sample streaming in the sense that sample streaming is the reason the sample buffer is divided into blocks and has such a complicated lifecycle. Prior to sample streaming, the sample buffer was a flat circular buffer and it was managed by an atomic increment of the cursor. If sample streaming is removed, we can return to that implementation.
Crashed once again:
../../runtime/vm/profiler.cc: 150: error: expected: sample_buffer_ != nullptr
===== CRASH =====
si_signo=Segmentation fault(11), si_code=SEGV_MAPERR(1), si_addr=0xababb1ab
Aborting reentrant request for stack trace.
-- BEGIN REPRODUCE --
DART SDK REVISION:
dart runtime/tools/dartfuzz/dartfuzz.dart --fp --no-ffi --no-flat --seed 2581978505 fuzz.dart
-- RUN 1 --
out/ReleaseX64C/dart --profiler --profile_vm=true --runtime_allocate_spill_tlab --no_concurrent_mark --old_gen_heap_size=128 /b/s/w/itj6ma63q9/dart_fuzzXSFATF/fuzz.dart
-- RUN 2 --
out/DebugX64C/dart --profiler --max_profile_depth=106 --gc_at_throw --verify_after_marking --old_gen_heap_size=128 /b/s/w/itj6ma63q9/dart_fuzzXSFATF/fuzz.dart
-- END REPRODUCE --
Still failing:
../../runtime/vm/profiler.cc: 150: error: expected: sample_buffer_ != nullptr
===== CRASH =====
si_signo=Segmentation fault(11), si_code=SEGV_MAPERR(1), si_addr=0xababb1a3
Aborting reentrant request for stack trace.
-- BEGIN REPRODUCE --
DART SDK REVISION:
dart runtime/tools/dartfuzz/dartfuzz.dart --no-fp --no-ffi --flat --seed 4061912401 fuzz.dart
-- RUN 1 --
out/ReleaseSIMARM64/dart --profiler --inlining_callee_size_threshold=225 --no_guess_icdata_cid --old_gen_heap_size=128 /b/s/w/itoggkepcq/dart_fuzzTVKDWN/fuzz.dart
-- RUN 2 --
out/DebugX64/dart --profiler --profile_vm=false --profile_period=5520 --mark_when_idle --test_il_serialization --no_use_field_guards --old_gen_heap_size=128 /b/s/w/itoggkepcq/dart_fuzzTVKDWN/fuzz.dart
-- END REPRODUCE --
Again https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket/8704325355839192129/+/u/collect_shards/dartfuzz_-_generated_programs_shard_3/task_stdout_stderr:dartfuzz-_generated_programs_shard_3
@mraleph , @derekxu16 mentioned you wanted to keep block-based sample buffer that he wanted to remove to get rid of these crashes. Is it still the case?
Yes, that is correct. I want to keep block based sample buffer for https://dart-review.googlesource.com/c/sdk/+/426220
Furthermore, I have few theories on how this specific race happens. I was planning to take a look at this soon.
I posted a TSAN reproduction on another issue, by the way: https://github.com/dart-lang/sdk/issues/61101#issuecomment-3184635737.