Sporadic unresolved test failures: `assert isinstance(result, tuple)`

[StephanTLavavej]

In the CI run https://dev.azure.com/vclibs/STL/_build/results?buildId=15888&view=results I noticed an unusual sporadic failure:

1: Unresolved Tests (1):
1:   libc++ :: std/input.output/string.streams/stringbuf/stringbuf.members/view.pass.cpp:2

Exception during script execution:
Traceback (most recent call last):
  File "D:\a/_work/1/s/llvm-project/llvm\utils\lit\lit\worker.py", line 77, in _execute_test_handle_errors
    return _adapt_result(result)
           ^^^^^^^^^^^^^^^^^^^^^
  File "D:\a/_work/1/s/llvm-project/llvm\utils\lit\lit\worker.py", line 92, in _adapt_result
    assert isinstance(result, tuple)
AssertionError

I don't know what happened here. It looks like microsoft/STL machinery returned a non-tuple to llvm-project machinery that was expecting a tuple.

[cpplearner]

I got the same error with the std/utilities/format/format.functions/escaped_output.ascii.pass.cpp test.

stl-lit.py: D:\STL\tests\utils\stl\test\format.py:250: error: UnicodeDecodeError('gbk', b"Assertion failure: out == expected D:\\STL\\llvm-project\\libcxx\\test\\std\\utilities\\format\\format.functions\\escaped_output.ascii.pass.cpp 46\r\n\r\nFormat string   {:?}\r\nExpected output '\x80'\r\nActual output   '\\x{80}'\r\nabort() has been called", 198, 199, 'illegal multibyte sequence') at D:\STL\tests\utils\stl\test\format.py:231
UNRESOLVED: libc++ :: std/utilities/format/format.functions/escaped_output.ascii.pass.cpp:1 (16 of 63)
******************** TEST 'libc++ :: std/utilities/format/format.functions/escaped_output.ascii.pass.cpp:1' FAILED ********************
Exception during script execution:
Traceback (most recent call last):
  File "D:\STL/llvm-project/llvm\utils\lit\lit\worker.py", line 77, in _execute_test_handle_errors
    return _adapt_result(result)
           ^^^^^^^^^^^^^^^^^^^^^
  File "D:\STL/llvm-project/llvm\utils\lit\lit\worker.py", line 92, in _adapt_result
    assert isinstance(result, tuple)
AssertionError


********************

It seems that STLTestFormat.execute returns None when an exception has been raised, but llvm's _adapt_result expects it to return something meaningful (either lit.Test.Result or tuple).

[StephanTLavavej]

(Transferred this comment's info to the issue above.)

[frederick-vs-ja]

[ ] What's causing this _CrtIsValidHeapPointer assertion, which is the underlying spurious failure?

I believe this is #4268. (The public_setg function calls setg, and then changes the return value of eback, which is unfortunately depended by MSVC STL.)

The underlying failure is actually stable, which generally causes FAIL.

It seems that the exception essentially comes from here: https://github.com/microsoft/STL/blob/202e3820d6390505855ce9a58a4bcee4f28feacd/tests/utils/stl/util.py#L74-L81

Perhaps the UnicodeDecodeError exception was triggered by particular data pattern in the corrupted heap. Such data seemed directly included in the output of Clang.

[StephanTLavavej]

Changing a runtime test to contain this:

// Copyright (c) Microsoft Corporation.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception

#include <cstdio>
using namespace std;

int main() {
    puts("HEAP CORRUPTION DETECTED: before \x8d");
}

caused it to reliably emit:

code: UNRESOLVED
name: std :: tests/P0586R2_integer_comparison:04
output: UnicodeDecodeError('charmap', b'HEAP CORRUPTION DETECTED: before \x8d\r\n', 33, 34, 'character maps to <undefined>') at D:\GitHub\STL\tests\utils\stl\test\format.py:231

I can fix this.

I'm still not sure exactly how the heap corruption is causing the UCRT to get confused when it tries to print the block type, but that's not something we can do anything about (we can fix iostreams someday, and we can fix the Python now).