aws-sdk-cpp icon indicating copy to clipboard operation
aws-sdk-cpp copied to clipboard
verified publisher icon aws

AddressSanitizer: heap-use-after-free during shutdown of Logger

Open bjosv opened this issue 2 months ago • 4 comments

Describe the bug

During shutdown of the CRT; if a thread takes longer time to respond than expected we might get a use after free. This can be seen when running a build using the address sanitizer (-DENABLE_ADDRESS_SANITIZER=ON, see logs below)

The issue seems to be predicted in the following comment: https://github.com/aws/aws-sdk-cpp/pull/2885/files#r1567933028

Regression Issue

  • [ ] Select this option if this issue appears to be a regression.

Expected Behavior

Not intermittently having use after free issues.

Current Behavior

Here are an output from a run when SUMMARY: AddressSanitizer: heap-use-after-free is hit during shutdown.

Aws::Utils::Logging::s_aws_logger_redirect_get_log_level(aws_logger*, unsigned int) /path/aws-sdk-cpp/src/aws-cpp-sdk-core/source/utils/logging/CRTLogging.cpp:59:42
    #1 0x7f2a17209f67 in s_destroy /path/aws-c-io/source/linux/epoll_event_loop.c:233:5
    #2 0x7f2a171e87de in s_aws_event_loop_group_shutdown_sync /path/aws-c-io/source/event_loop.c:43:13
    #3 0x7f2a171ebde5 in s_event_loop_destroy_async_thread_fn /path/aws-c-io/source/event_loop.c:55:5
    #4 0x7f2a18a53bfb in thread_fn /path/aws-c-common/source/posix/thread.c:177:5
    #5 0x7f2a1bcfdaf6  (/usr/local/lib/clang/21/lib/x86_64-unknown-linux-gnu/libclang_rt.asan.so+0x147af6)
    #6 0x7f2a19b8af6b in start_thread /usr/src/debug/glibc-2.38-150600.14.37.1.x86_64/nptl/pthread_create.c:444:8
    #7 0x7f2a19c12387 in __GI___clone3 /usr/src/debug/glibc-2.38-150600.14.37.1.x86_64/misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

0x7b5a153e2120 is located 16 bytes inside of 24-byte region [0x7b5a153e2110,0x7b5a153e2128)
freed by thread T0 here:
    #0 0x7f2a1bd1107d in operator delete(void*) (/usr/local/lib/clang/21/lib/x86_64-unknown-linux-gnu/libclang_rt.asan.so+0x15b07d)
    #1 0x7f2a19b3dba0 in __cxa_finalize /usr/src/debug/glibc-2.38-150600.14.37.1.x86_64/stdlib/cxa_finalize.c:82:6

previously allocated by thread T1 here:
    #0 0x7f2a1bd1081d in operator new(unsigned long) (/usr/local/lib/clang/21/lib/x86_64-unknown-linux-gnu/libclang_rt.asan.so+0x15a81d)
    #1 0x7f2a18dfe25e in std::__new_allocator<std::_Sp_counted_ptr_inplace<Aws::Utils::Logging::DefaultCRTLogSystem, std::allocator<Aws::Utils::Logging::DefaultCRTLogSystem>, (__gnu_cxx::_Lock_policy)2>>::allocate(unsigned long, void const*) /usr/local/bin/../lib/gcc/x86_64-linux-gnu/15.2.0/../../../../include/c++/15.2.0/bits/new_allocator.h:151:27
    #2 0x7f2a18dfe25e in std::allocator_traits<std::allocator<std::_Sp_counted_ptr_inplace<Aws::Utils::Logging::DefaultCRTLogSystem, std::allocator<Aws::Utils::Logging::DefaultCRTLogSystem>, (__gnu_cxx::_Lock_policy)2>>>::allocate(std::allocator<std::_Sp_counted_ptr_inplace<Aws::Utils::Logging::DefaultCRTLogSystem, std::allocator<Aws::Utils::Logging::DefaultCRTLogSystem>, (__gnu_cxx::_Lock_policy)2>>&, unsigned long) /usr/local/bin/../lib/gcc/x86_64-linux-gnu/15.2.0/../../../../include/c++/15.2.0/bits/alloc_traits.h:614:20
    #3 0x7f2a18dfe25e in std::__allocated_ptr<std::allocator<std::_Sp_counted_ptr_inplace<Aws::Utils::Logging::DefaultCRTLogSystem, std::allocator<Aws::Utils::Logging::DefaultCRTLogSystem>, (__gnu_cxx::_Lock_policy)2>>> std::__allocate_guarded<std::allocator<std::_Sp_counted_ptr_inplace<Aws::Utils::Logging::DefaultCRTLogSystem, std::allocator<Aws::Utils::Logging::DefaultCRTLogSystem>, (__gnu_cxx::_Lock_policy)2>>>(std::allocator<std::_Sp_counted_ptr_inplace<Aws::Utils::Logging::DefaultCRTLogSystem, std::allocator<Aws::Utils::Logging::DefaultCRTLogSystem>, (__gnu_cxx::_Lock_policy)2>>&) /usr/local/bin/../lib/gcc/x86_64-linux-gnu/15.2.0/../../../../include/c++/15.2.0/bits/allocated_ptr.h:102:21
    #4 0x7f2a18dfe25e in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<Aws::Utils::Logging::DefaultCRTLogSystem, std::allocator<Aws::Utils::Logging::DefaultCRTLogSystem>, Aws::Utils::Logging::LogLevel const&>(Aws::Utils::Logging::DefaultCRTLogSystem*&, std::_Sp_alloc_shared_tag<std::allocator<Aws::Utils::Logging::DefaultCRTLogSystem>>, Aws::Utils::Logging::LogLevel const&) /usr/local/bin/../lib/gcc/x86_64-linux-gnu/15.2.0/../../../../include/c++/15.2.0/bits/shared_ptr_base.h:967:19
    #5 0x7f2a18dfe25e in std::__shared_ptr<Aws::Utils::Logging::DefaultCRTLogSystem, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<Aws::Utils::Logging::DefaultCRTLogSystem>, Aws::Utils::Logging::LogLevel const&>(std::_Sp_alloc_shared_tag<std::allocator<Aws::Utils::Logging::DefaultCRTLogSystem>>, Aws::Utils::Logging::LogLevel const&) /usr/local/bin/../lib/gcc/x86_64-linux-gnu/15.2.0/../../../../include/c++/15.2.0/bits/shared_ptr_base.h:1719:14
    #6 0x7f2a18dfe25e in std::shared_ptr<Aws::Utils::Logging::DefaultCRTLogSystem>::shared_ptr<std::allocator<Aws::Utils::Logging::DefaultCRTLogSystem>, Aws::Utils::Logging::LogLevel const&>(std::_Sp_alloc_shared_tag<std::allocator<Aws::Utils::Logging::DefaultCRTLogSystem>>, Aws::Utils::Logging::LogLevel const&) /usr/local/bin/../lib/gcc/x86_64-linux-gnu/15.2.0/../../../../include/c++/15.2.0/bits/shared_ptr.h:463:4
    #7 0x7f2a18dfe25e in std::shared_ptr<std::enable_if<!is_array<Aws::Utils::Logging::DefaultCRTLogSystem>::value, Aws::Utils::Logging::DefaultCRTLogSystem>::type> std::allocate_shared<Aws::Utils::Logging::DefaultCRTLogSystem, std::allocator<Aws::Utils::Logging::DefaultCRTLogSystem>, Aws::Utils::Logging::LogLevel const&>(std::allocator<Aws::Utils::Logging::DefaultCRTLogSystem> const&, Aws::Utils::Logging::LogLevel const&) /usr/local/bin/../lib/gcc/x86_64-linux-gnu/15.2.0/../../../../include/c++/15.2.0/bits/shared_ptr.h:990:14
    #8 0x7f2a18dfe25e in std::shared_ptr<Aws::Utils::Logging::DefaultCRTLogSystem> Aws::MakeShared<Aws::Utils::Logging::DefaultCRTLogSystem, Aws::Utils::Logging::LogLevel const&>(char const*, Aws::Utils::Logging::LogLevel const&) /path/aws-sdk-cpp/src/aws-cpp-sdk-core/include/aws/core/utils/memory/stl/AWSAllocator.h:117:16
    #9 0x7f2a18dfe25e in Aws::InitAPI(Aws::SDKOptions const&) /path/aws-sdk-cpp/src/aws-cpp-sdk-core/source/Aws.cpp:72:25
    ...

Thread T19 created by T0 here:
    #0 0x7f2a1bce42f1 in pthread_create (/usr/local/lib/clang/21/lib/x86_64-unknown-linux-gnu/libclang_rt.asan.so+0x12e2f1)
    #1 0x7f2a18a52593 in aws_thread_launch /path/aws-c-common/source/posix/thread.c:352:19
    #2 0x7f2a171e823e in s_aws_event_loop_group_shutdown_async /path/aws-c-io/source/event_loop.c:75:5
    #3 0x7f2a18a5a96c in aws_ref_count_release /path/aws-c-common/source/ref_count.c:29:9
    #4 0x7f2a171f89a4 in s_cleanup_default_resolver /path/aws-c-io/./3pp/sources/aws-c-io/source/host_resolver.c:330:5
    #5 0x7f2a18a5a96c in aws_ref_count_release /path/aws-c-common/source/ref_count.c:29:9
    #6 0x7f2a171d3b23 in s_client_bootstrap_destroy_impl /path/aws-c-io/source/channel_bootstrap.c:31:5
    #7 0x7f2a18a5a96c in aws_ref_count_release /path/aws-c-comm on/source/ref_count.c:29:9
    #8 0x7f2a17de36c8 in Aws::Crt::Io::ClientBootstrap::~ClientBootstrap() /path/aws-crt-cpp/source/io/Bootstrap.cpp:91:21
    #9 0x7f2a1b92db2e in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /usr/local/bin/../lib/gcc/x86_64-linux-gnu/15.2.0/../../../../include/c++/15.2.0/bits/shared_ptr_base.h:345:8
    #10 0x7f2a1b92db2e in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/local/bin/../lib/gcc/x86_64-linux-gnu/15.2.0/../../../../include/c++/15.2.0/bits/shared_ptr_base.h:1069:11
    #11 0x7f2a19b3dba0 in __cxa_finalize /usr/src/debug/glibc-2.38-150600.14.37.1.x86_64/stdlib/cxa_finalize.c:82:6

Thread T1 created by T0 here:
    #0 0x7f2a1bce42f1 in pthread_create (/usr/local/lib/clang/21/lib/x86_64-unknown-linux-gnu/libclang_rt.asan.so+0x12e2f1)
    #1 0x7f2a1b40b772 in mt_thread_create /path/mt_thread.c:38:38
    ...
    #4 0x55e5796c45a9 in main /path/main.c:512:5
    #5 0x7f2a19b24e6b in __libc_start_call_main /usr/src/debug/glibc-2.38-150600.14.37.1.x86_64/csu/../sysdeps/nptl/libc_start_call_main.h:58:16

SUMMARY: AddressSanitizer: heap-use-after-free /path/aws-sdk-cpp/src/aws-cpp-sdk-core/source/utils/logging/CRTLogging.cpp:59:42 in Aws::Utils::Logging::s_aws_logger_redirect_get_log_level(aws_logger*, unsigned int)

Reproduction Steps

Will investigate if an existing aws-sdk-cpp test can be modified to reproduce.

Possible Solution

No response

Additional Information/Context

No response

AWS CPP SDK version used

1.11.479 and later

Compiler and Version used

clang 21

Operating System and version

Linux, Ubuntu 24.04

bjosv avatar Nov 03 '25 22:11 bjosv

Thanks for bringing this up. This is on our radar and something we will address in the future, as the fix/refactor is not trivial to resolve this race condition. In the meantime, is it possible for you to wait for threads prior to shutting down the SDK?

sbaluja avatar Nov 12 '25 16:11 sbaluja

Thanks for bringing this up. This is on our radar and something we will address in the future, as the fix/refactor is not trivial to resolve this race condition. In the meantime, is it possible for you to wait for threads prior to shutting down the SDK?

Maybe I don't catch your point, but the logic is in SDK itself, do you mean we make a change in the SDK to wait more time before finally shutting down CRTLogging?

TeddyX95 avatar Nov 14 '25 04:11 TeddyX95

If this issue is impacting you I'm assuming your application is multithreaded and sharing a single SDK instance. Unless you're initializing and shutting down the SDK inside each thread (which would severely impact performance), you likely have a main thread handling that lifecycle. My question is whether it's possible in your setup to wait for all worker threads to finish before shutting down the SDK, as a temporary workaround for this race condition. If this doesn't apply to your scenario, feel free to disregard; we will look at this issue in the future.

sbaluja avatar Nov 14 '25 05:11 sbaluja

If this issue is impacting you I'm assuming your application is multithreaded and sharing a single SDK instance. Unless you're initializing and shutting down the SDK inside each thread (which would severely impact performance), you likely have a main thread handling that lifecycle. My question is whether it's possible in your setup to wait for all worker threads to finish before shutting down the SDK, as a temporary workaround for this race condition. If this doesn't apply to your scenario, feel free to disregard; we will look at this issue in the future.

Hi, I have tried your suggested workaround and it works. Thanks very much for the support.

TeddyX95 avatar Nov 27 '25 06:11 TeddyX95