STL icon indicating copy to clipboard operation
STL copied to clipboard

Sporadic STL-ASan-CI x86 AVs

Open CaseyCarter opened this issue 1 year ago • 2 comments
trafficstars

In STL-ASan-CI runs, seemingly random x86 test cases fail with exit code 3221225477 (0xC0000005) indicating an Access Violation. Let's track sporadic AVs here and see if there's a pattern.

Note that these invalid accesses are not caught and reported by the ASan runtime, so they are presumably uninstrumented. Together with the fact that this is ASan-only and x86-only, I suspect a race in ASan setup or teardown.

This is on no way an exhaustive list: we've missed tracking some, and nearly given up after 2024-08 because they are so frequent. In any case, there seems to be no pattern. It's probably some kind of shutdown race.

Date Toolset PR Test Run
2024-01-13 17.9p2 #4304 (CI) P0896R4_ranges_alg_remove_copy
2024-01-15 17.9p2 #4313 GH_001123_random_cast_out_of_range
2024-01-28 17.9p2 #4341 (CI) P0768R1_spaceship_operator
2024-03-23 17.10p2 #4493 (CI) VSO_0180466_algorithm_overhauls
2024-04-10 17.10p2 #4576 tr1/tests/new
2024-04-13 17.10p3 #4578 (CI) P1023R0_constexpr_for_array_comparisons
2024-04-23 17.10p4 #4599 (CI) P0448R4_spanstream
2024-04-24 17.10p4 #4599 (CI) Dev10_860421_deque_push_back_pop_front
2024-05-03 17.10p4 #4652 libcxx range.iota.view/iterator/ctor.default.pass.cpp
2024-05-20 17.10p4 #4677 (CI) P1135R6_atomic_wait_vista
2024-05-24 17.11p1 #4685 (CI) P2374R4_checked_arithmetic_operations + 2 more
2024-06-12 17.11p2 #4725 GH_001010_filesystem_error_encoding
2024-06-26 17.11p2 #4742 (CI) P2278R4_basic_const_iterator + 1 more
2024-07-09 17.11p3 #4824 libcxx func.bind.bind/invoke_rvalue.pass.cpp + 1 more
2024-07-12 17.11p3 #4819 (CI) Dev09_199123_tr1_mem_fun_abstract_classes
2024-07-21 17.11p3 #4844 (CI) P0355R7_calendars_and_time_zones_dates
2024-07-26 17.11p3 #4862 (CI) Dev10_445289_make_shared + 1 more
2024-07-31 17.11p3 #4870 (CI) P0811R3_midpoint_lerp + 1 more
2024-08-14 17.12p1 #4889 libcxx iterator.range/begin-end.array.pass.cpp + 2 more
2024-08-24 17.12p1 #4910 libcxx char.traits.specializations.wchar_t/eq.pass.cpp
2025-03-11 17.14p2 #5335 Dev11_0000000_quoted
2025-04-17 17.14p3 #5410 P0896R4_views_all
2025-04-30 17.14p3 #5436 (CI) P2286R8_text_formatting_escaping_utf8

CaseyCarter avatar Jan 16 '24 20:01 CaseyCarter

Added a new failure from March 23 to the table. Given that these failures are all x86-specific, I now suspect the root cause is the ASan-x86-specific OPARGPLACE backend bug VSO-1974914. This issue causes thousands of checked-compiler assertion failures in internal STL ASan-x86 testing that would likely manifest as Silent Bad Codegen with a release compiler.

The fix for VSO-1974914 will likely ship in 17.11.p1. We should close this issue after updating our CI compiler to 17.11p1, and we'll reopen if we continue to see sporadic failures.

CaseyCarter avatar Apr 04 '24 19:04 CaseyCarter

I think I'm still seeing this with the 17.11 Preview 1 toolset update.

StephanTLavavej avatar May 22 '24 06:05 StephanTLavavej