The MSVC team recently test ONNX project failed to run test on release configuration on windows_x64.

[Vincent9802]

trafficstars

Hi All,

ONNX failed to run test on release configuration with MSVC on windows x64. Could you please help look at this issue or provide some workarounds? Thanks in advance.

Error message

test.log.txt

==45536==ERROR: AddressSanitizer: container-overflow on address 0x00743492e787 at pc 0x7ffc4b3767bb bp 0x00743492e560 sp 0x00743492dce0 READ of size 16 at 0x00743492e787 thread T0 #0 0x7ffc4b3767e8 (C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX64\x64\clang_rt.asan_dynamic-x86_64.dll+0x1800467e8) #1 0x7ffc4cba1913 (F:\gitP\onnx\onnx\build_amd64\Release\libprotobuf.dll+0x180001913) #2 0x7ffc4cba920c (F:\gitP\onnx\onnx\build_amd64\Release\libprotobuf.dll+0x18000920c) #3 0x7ff768d35dcc in onnx::OnnxParser::Parse(class std::basic_string<char, struct std::char_traits, class std::allocator>, class onnx::GraphProto &) (F:\gitP\onnx\onnx\build_amd64\Release\onnx_gtests.exe+0x1400f5dcc) #4 0x7ff768d3842c in onnx::OnnxParser::Parse(class onnx::GraphProto &) (F:\gitP\onnx\onnx\build_amd64\Release\onnx_gtests.exe+0x1400f842c) #5 0x7ff768c5ef24 in onnx::Test::RunDataPropagation(char const *, int) (F:\gitP\onnx\onnx\build_amd64\Release\onnx_gtests.exe+0x14001ef24) #6 0x7ff768c4948e in onnx::Test::DataPropagationImplTest_ShapeTest_Test::TestBody(void) (F:\gitP\onnx\onnx\build_amd64\Release\onnx_gtests.exe+0x14000948e) #7 0x7ff768cd8fe7 in testing::internal::HandleSehExceptionsInMethodIfSupported<class testing::Test, void>(class testing::Test , void (__cdecl testing::Test::)(void), char const *) F:\gitP\onnx\onnx\build_amd64\googletest\src\googletest\googletest\src\gtest.cc:2443 #8 0x7ff768cd8b3b in testing::internal::HandleExceptionsInMethodIfSupported<class testing::Test, void>(class testing::Test , void (__cdecl testing::Test::)(void), char const *) F:\gitP\onnx\onnx\build_amd64\googletest\src\googletest\googletest\src\gtest.cc:2494 #9 0x7ff768d16922 in testing::TestInfo::Run(void) F:\gitP\onnx\onnx\build_amd64\googletest\src\googletest\googletest\src\gtest.cc:2713 #10 0x7ff768d1655f in testing::TestCase::Run(void) F:\gitP\onnx\onnx\build_amd64\googletest\src\googletest\googletest\src\gtest.cc:2836 #11 0x7ff768d1721e in testing::internal::UnitTestImpl::RunAllTests(void) F:\gitP\onnx\onnx\build_amd64\googletest\src\googletest\googletest\src\gtest.cc:5249 #12 0x7ff768cd9107 in testing::internal::HandleSehExceptionsInMethodIfSupported<class testing::internal::UnitTestImpl, bool>(class testing::internal::UnitTestImpl , bool (__cdecl testing::internal::UnitTestImpl::)(void), char const *) F:\gitP\onnx\onnx\build_amd64\googletest\src\googletest\googletest\src\gtest.cc:2443 #13 0x7ff768cd8f39 in testing::internal::HandleExceptionsInMethodIfSupported<class testing::internal::UnitTestImpl, bool>(class testing::internal::UnitTestImpl , bool (__cdecl testing::internal::UnitTestImpl::)(void), char const *) F:\gitP\onnx\onnx\build_amd64\googletest\src\googletest\googletest\src\gtest.cc:2494 #14 0x7ff768d16cba in testing::UnitTest::Run(void) F:\gitP\onnx\onnx\build_amd64\googletest\src\googletest\googletest\src\gtest.cc:4858 #15 0x7ff768cd49e8 in main (F:\gitP\onnx\onnx\build_amd64\Release\onnx_gtests.exe+0x1400949e8) #16 0x7ff768f21233 in __scrt_common_main_seh D:\a01_work\2\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288 #17 0x7ffc7e2e7973 (C:\Windows\System32\KERNEL32.DLL+0x180017973) #18 0x7ffc8104a2f0 (C:\Windows\SYSTEM32\ntdll.dll+0x18005a2f0)

Address 0x00743492e787 is located in stack of thread T0 at offset 119 in frame #0 0x7ff768d382af in onnx::OnnxParser::Parse(class onnx::GraphProto &) (F:\gitP\onnx\onnx\build_amd64\Release\onnx_gtests.exe+0x1400f82af)

This frame has 3 object(s): [32, 64) 'id' [48, 56) 'compiler temporary' [64, 96) 'compiler temporary' <== Memory access at offset 119 overflows this variable HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork (longjmp, SEH and C++ exceptions are supported) HINT: if you don't care about these errors you may set ASAN_OPTIONS=detect_container_overflow=0. If you suspect a false positive see also: https://github.com/google/sanitizers/wiki/AddressSanitizerContainerOverflow. SUMMARY: AddressSanitizer: container-overflow (C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX64\x64\clang_rt.asan_dynamic-x86_64.dll+0x1800467e8) Shadow bytes around the buggy address: 0x022d51da5ca0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x022d51da5cb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x022d51da5cc0: f1 f1 f1 f1 f8 f2 f8 f2 f8 f2 f8 f2 f8 f3 f3 f3 0x022d51da5cd0: f3 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x022d51da5ce0: 00 00 f1 f1 f1 f1 07 fc 00 00 f2 f2 f2 f2 f8 f2 =>0x022d51da5cf0:[07]fc 00 00 f3 f3 f3 f3 00 00 00 00 00 00 00 00 0x022d51da5d00: 00 00 00 00 f1 f1 f1 f1 00 00 00 00 00 00 00 00 0x022d51da5d10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x022d51da5d20: 00 00 f2 f2 f2 f2 00 00 00 00 00 00 00 00 f2 f2 0x022d51da5d30: f2 f2 00 00 00 00 00 00 00 00 f2 f2 f2 f2 00 00 0x022d51da5d40: 00 00 00 00 00 00 f2 f2 f2 f2 00 00 00 00 00 00 Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==45536==ABORTING

Reproduction instructions

1. git clone https://github.com/onnx/onnx.git F:\gitP\onnx
2. git clone https://github.com/Microsoft/vcpkg F:\gitP\Microsoft
3. set VSCMD_SKIP_SENDTELEMETRY=1 & "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\Tools\VsDevCmd.bat" -host_arch=amd64 -arch=amd64 & set CL= /fsanitize=address /GS- /wd5072 & set LINK= /InferASanLibs /incremental:no /debug
4. cd F:\gitP\Microsoft\vcpkg
5. bootstrap-vcpkg.bat 2>&1
6. vcpkg.exe install --recurse Protobuf --triplet x64-windows --clean-after-build 2>&1
7. mkdir&cd F:\gitP\onnx\onnx\build_amd64
8. cmake -G "Visual Studio 16 2019" -A x64 -DCMAKE_SYSTEM_VERSION=10.0.18362.0 -DCMAKE_BUILD_TYPE=Release -DONNX_USE_LITE_PROTO=ON -DONNX_BUILD_TESTS=ON -DCMAKE_TOOLCHAIN_FILE=F:\gitP\Microsoft\vcpkg\scripts\buildsystems\vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x64-windows .. 2>&1
9. msbuild /m /p:Platform=x64 /p:Configuration=Release onnx.sln /t:Rebuild 2>&1
10. cd F:\gitP\onnx\onnx\build_amd64\Release
11. onnx_gtests.exe 2>&1

Expected behavior

test pass

System information

• OS Platform and Distribution Window10
• ONNX version 1.12.0
• Python version: 3.9
• CMake version: 3.22.0
• Protobuf version: 3.19.4
• Visual Studio version (if applicable): Visual Studio 2019

[jcwchen]

Hi @Vincent9802, Thanks for providing the details, but I cannot repro this error both in my local and CI (I don't have the exact version as yours btw). The error seems happened here, but the code looks fine to me.

I guess perhaps there might be some compiler config difference? It is possible that it's a false positive? I saw there is a message saying that HINT: if you don't care about these errors you may set ASAN_OPTIONS=detect_container_overflow=0., which might be a possible workaround for now.

cc @gramalingam since the error seems happen during onnx::OnnxParser::Parse. Did you see similar error before?

[gramalingam]

This seems to be produced by a address-sanitizer check? Can you give us some context about this checker? One of the links suggests that it could produce false-positives too. How does one validate if this is a false-positive or not?

[gramalingam]

IIUC the stack, the error happens in the protobuf library when called from this function: https://github.com/onnx/onnx/blob/b064f0a88cbe2fb75081dc8d25adde942ad9a916/onnx/defs/parser.cc#L511 I don't see any obvious error in the protobuf usage that would cause such a violation (unless it is in the protobuf implementation).

[Vincent9802]

@jcwchen @gramalingam

Thanks for your reply and suggestion. This error was found when compiling this project in the latest development version of VS, and is only reproduced in ASAN mode. I apologize for not clarifying this issue, I am trying to set the ASAN option to clear the error. To reproduce, you need to add the setting set CL= /fsanitize=address /GS- /wd5072 & set LINK= /InferASanLibs /incremental:no /debug in the VS x64 command box.

[justinchuby]

Tracked by #4939