aws-sdk-cpp Memory leak in functions Aws::DynamoDB::DynamoDBClient::xxxxxCallable

Confirm by changing [ ] to [x] below to ensure that it's a bug:

[ x] I've gone though Developer Guide and API reference
[x ] I've searched for previous similar issues and didn't find any solution

Describe the bug Attempt to write multiple items to dynamodb using BatchWriteItemCallable results in memory leaks SIMILAR LEAKAGE OCCURS WITH QueryCallable()

SDK version number 1.9.118

Platform/OS/Hardware/Device AWS Linux ubuntu

To Reproduce (observed behavior) When CALLABLE_VERSION is set to 1 below, memory leak occurs; when it is set to 0 no memory leak.

/**
@file Main.cpp
*/
//#define DOCTEST_CONFIG_IMPLEMENT
//#include "doctest/doctest.h"

#include <iostream>
#include <thread>
#include <csignal>

#include <aws/core/Aws.h>
#include <aws/core/utils/Outcome.h>
#include <aws/dynamodb/DynamoDBClient.h>
#include <aws/dynamodb/model/AttributeDefinition.h>
#include <aws/dynamodb/model/PutItemRequest.h>
#include <aws/dynamodb/model/PutRequest.h>
#include <aws/dynamodb/model/PutItemResult.h>
#include <aws/dynamodb/model/GetItemRequest.h>
#include <aws/dynamodb/model/GetItemResult.h>
#include <aws/dynamodb/model/QueryRequest.h>
#include <aws/dynamodb/model/QueryResult.h>
#include <aws/dynamodb/model/ListTablesRequest.h>
#include <aws/dynamodb/model/ListTablesResult.h>
#include <aws/dynamodb/model/BatchWriteItemRequest.h>
#include <aws/dynamodb/model/BatchWriteItemResult.h>

Aws::DynamoDB::Model::PutRequest createSingleItem();

int main(int argc, char* argv[])
{

	using namespace std;
    cout << "Program is starting ...\n";

    // ---- Disable the SIGPIPE signal
    std::signal(SIGPIPE,SIG_IGN);

    // --- AWS init
    #define CALLABLE_VERSION 1
    Aws::SDKOptions options;
    Aws::InitAPI(options);
    {
        using namespace Aws::DynamoDB::Model;
        Aws::Vector<Aws::DynamoDB::Model::WriteRequest> pendingRequests;
        Aws::DynamoDB::DynamoDBClient dynamoClient;
        // Number of items in the batch
        constexpr size_t nbrRequests{1};
        // Create the list of PutRequest
        for (size_t i{0}; i < nbrRequests; ++i)
        {
            auto pir = createSingleItem();
            pendingRequests.push_back(std::move(WriteRequest().WithPutRequest(std::move(pir))));
        }

        // Time to send the whole thing to the servera
        BatchWriteItemRequest bwir;
        std::string tableName = "ondemand_test_vessels";
        Aws::Map<Aws::String,Aws::Vector<WriteRequest>> requestMap;
        requestMap.emplace(tableName,std::move(pendingRequests));
        bwir.WithRequestItems(std::move(requestMap));

        #if CALLABLE_VERSION
            auto fut = dynamoClient.BatchWriteItemCallable(bwir);
            fut.get();
        #else
            dynamoClient.BatchWriteItem(bwir);
        #endif
        pendingRequests.clear();
    }
    Aws::ShutdownAPI(options);

    cout << "Program (Main) is terminating\n";
} // end main


#include <random>

Aws::DynamoDB::Model::PutRequest createSingleItem()
{
    using namespace Aws::DynamoDB;
    using namespace std::string_literals;

    // ---- Create the PutRequest
    Model::PutRequest pir;
    Model::AttributeValue av;
    static std::default_random_engine dre;
    static std::uniform_int_distribution<unsigned> mmsi_rand(1,999999999u);
    static std::uniform_int_distribution<unsigned long> sort_rand(1,9999999999u);
    // ---- Create the PutRequest
    // Partition key
    av.SetS("DYN"s + std::to_string(mmsi_rand(dre)));
    pir.AddItem("PK",std::move(av));
    // Sort key
    av.SetS(std::to_string(sort_rand(dre)));
    pir.AddItem("SK",std::move(av));
    // Add a random value attribute
    av.SetN(1590970980);
    pir.AddItem("test", std::move(av));

    return pir;
}

Expected behavior No memory leak.

Logs/output None

options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Trace;
Aws::InitAPI(options)

Additional context The memory leak as detected by Valgrind are as follows:

==306190== Memcheck, a memory error detector
==306190== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==306190== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info
==306190== Command: ./RtDbWriter.execute --no-run=true
==306190== 
Program is starting ...
Program (Main) is terminating
==306190== 
==306190== HEAP SUMMARY:
==306190==     in use at exit: 6,520 bytes in 25 blocks
==306190==   total heap usage: 80,468 allocs, 80,443 frees, 7,659,306 bytes allocated
==306190== 
==306190== 432 (168 direct, 264 indirect) bytes in 1 blocks are definitely lost in loss record 9 of 12
==306190==    at 0x483577F: malloc (vg_replace_malloc.c:299)
==306190==    by 0x56B4558: CRYPTO_zalloc (in /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1)
==306190==    by 0x5312994: s2n_drbg_instantiate (s2n_drbg.c:155)
==306190==    by 0x5309CF5: s2n_defend_if_forked (s2n_random.c:143)
==306190==    by 0x5309E4F: s2n_get_private_random_data (s2n_random.c:175)
==306190==    by 0x530A1D8: s2n_openssl_compat_rand (s2n_random.c:295)
==306190==    by 0x51FCA80: Aws::Utils::Crypto::SecureRandomBytes_OpenSSLImpl::GetBytes(unsigned char*, unsigned long) (CryptoImpl.cpp:142)
==306190==    by 0x51B9DFA: Aws::Utils::UUID::RandomUUID() (UUID.cpp:79)
==306190==    by 0x51E71A9: Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const (AWSClient.cpp:235)
==306190==    by 0x51EC1E6: Aws::Client::AWSJsonClient::MakeRequest(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const (AWSClient.cpp:910)
==306190==    by 0x4C2DD23: Aws::DynamoDB::DynamoDBClient::BatchWriteItem(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const (DynamoDBClient.cpp:292)
==306190==    by 0x4C2DDAA: Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}::operator()() const (DynamoDBClient.cpp:297)
==306190==    by 0x4CF135E: Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> std::__invoke_impl<Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError>, Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}&>(std::__invoke_other, Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}&) (invoke.h:60)
==306190==    by 0x4CE9E7E: std::enable_if<is_invocable_r_v<Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError>, Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}&>, std::enable_if>::type std::__invoke_r<Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError>, Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}&>(Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError>&&, (Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}&)...) (invoke.h:115)
==306190==    by 0x4CE35D9: std::__future_base::_Task_state<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}, std::allocator<int>, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> ()>::_M_run()::{lambda()#1}::operator()() const (future:1425)
==306190==    by 0x4D04A53: std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}, std::allocator<int>, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> ()>::_M_run()::{lambda()#1}, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >::operator()() const (future:1342)
==306190==    by 0x4D0005F: std::unique_ptr<std::__future_base::_Result<Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >, std::__future_base::_Result_base::_Deleter> std::__invoke_impl<std::unique_ptr<std::__future_base::_Result<Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}, std::allocator<int>, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> ()>::_M_run()::{lambda()#1}, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >&>(std::__invoke_other, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}, std::allocator<int>, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> ()>::_M_run()::{lambda()#1}, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >&) (invoke.h:60)
==306190==    by 0x4CFA8F7: std::enable_if<is_invocable_r_v<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}, std::allocator<int>, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> ()>::_M_run()::{lambda()#1}, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >&>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}, std::allocator<int>, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> ()>::_M_run()::{lambda()#1}, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >&>::type std::__invoke_r<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}, std::allocator<int>, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> ()>::_M_run()::{lambda()#1}, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >&>(std::enable_if&&, (std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>&&)...) (invoke.h:113)
==306190==    by 0x4CF13E0: std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}, std::allocator<int>, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> ()>::_M_run()::{lambda()#1}, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> > >::_M_invoke(std::_Any_data const&) (std_function.h:292)
==306190==    by 0x4D1233E: std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const (std_function.h:622)
==306190==    by 0x4D05DBC: std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) (future:564)
==306190==    by 0x4D31F52: void std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (invoke.h:73)
==306190==    by 0x4D2355C: std::__invoke_result<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>::type std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (invoke.h:95)
==306190==    by 0x4D11FDB: void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}::operator()() const (mutex:717)
==306190==    by 0x4D1200E: void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::operator()() const (mutex:722)
==306190==    by 0x4D1201F: void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::_FUN() (mutex:722)
==306190==    by 0x5850996: __pthread_once_slow (pthread_once.c:116)
==306190==    by 0x4D04F9A: __gthread_once(int*, void (*)()) (gthr-default.h:700)
==306190==    by 0x4D120C9: void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (mutex:729)
==306190==    by 0x4D05A28: std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) (future:404)
==306190==    by 0x4CE363B: std::__future_base::_Task_state<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}, std::allocator<int>, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> ()>::_M_run() (future:1427)
==306190==    by 0x4D144BC: std::packaged_task<Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> ()>::operator()() (future:1560)
==306190==    by 0x4C2DDF1: Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2}::operator()() const (DynamoDBClient.cpp:298)
==306190==    by 0x4CBA3BE: void std::__invoke_impl<void, Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2}&>(std::__invoke_other, Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2}&) (invoke.h:60)
==306190==    by 0x4CB87D4: std::__invoke_result<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2}&>::type std::__invoke<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2}&>(std::__invoke_result&&, (Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2}&)...) (invoke.h:95)
==306190==    by 0x4CB7025: void std::_Bind<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2} ()>::__call<void>(std::tuple<>&&, std::_Index_tuple<>) (functional:416)
==306190==    by 0x4CB38E3: void std::_Bind<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2} ()>::operator()<, void>() (functional:499)
==306190==    by 0x4CA8F7D: void std::__invoke_impl<void, std::_Bind<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2} ()>&>(std::__invoke_other, std::_Bind<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2} ()>&) (invoke.h:60)
==306190==    by 0x4C9E13F: std::enable_if<is_invocable_r_v<void, std::_Bind<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2} ()>&>, std::enable_if>::type std::__invoke_r<void, std::_Bind<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2} ()>&>(void&&, (std::_Bind<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2} ()>&)...) (invoke.h:110)
==306190==    by 0x4C9469B: std::_Function_handler<void (), std::_Bind<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2} ()> >::_M_invoke(std::_Any_data const&) (std_function.h:291)
==306190==    by 0x51B4A19: std::function<void ()>::operator()() const (std_function.h:622)
==306190==    by 0x5203936: Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void ()>&&)::{lambda()#1}::operator()() const (Executor.cpp:18)
==306190==    by 0x52060D5: void std::__invoke_impl<void, Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void ()>&&)::{lambda()#1}>(std::__invoke_other, Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void ()>&&)::{lambda()#1}&&) (invoke.h:60)
==306190==    by 0x520607C: std::__invoke_result<Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void ()>&&)::{lambda()#1}>::type std::__invoke<Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void ()>&&)::{lambda()#1}>(std::__invoke_result&&, (Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void ()>&&)::{lambda()#1}&&)...) (invoke.h:95)
==306190==    by 0x5206029: void std::thread::_Invoker<std::tuple<Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void ()>&&)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (thread:264)
==306190==    by 0x5205FFD: std::thread::_Invoker<std::tuple<Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void ()>&&)::{lambda()#1}> >::operator()() (thread:271)
==306190==    by 0x5205FE1: std::thread::_State_impl<std::thread::_Invoker<std::tuple<Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void ()>&&)::{lambda()#1}> > >::_M_run() (thread:215)
==306190==    by 0x59BFBEF: ??? (in /usr/local/lib64/libstdc++.so.6.0.28)
==306190==    by 0x5848FA2: start_thread (pthread_create.c:486)
==306190==    by 0x5D564CE: clone (clone.S:95)
==306190== 
==306190== 432 (168 direct, 264 indirect) bytes in 1 blocks are definitely lost in loss record 10 of 12
==306190==    at 0x483577F: malloc (vg_replace_malloc.c:299)
==306190==    by 0x56B4558: CRYPTO_zalloc (in /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1)
==306190==    by 0x5312994: s2n_drbg_instantiate (s2n_drbg.c:155)
==306190==    by 0x5309D24: s2n_defend_if_forked (s2n_random.c:144)
==306190==    by 0x5309E4F: s2n_get_private_random_data (s2n_random.c:175)
==306190==    by 0x530A1D8: s2n_openssl_compat_rand (s2n_random.c:295)
==306190==    by 0x51FCA80: Aws::Utils::Crypto::SecureRandomBytes_OpenSSLImpl::GetBytes(unsigned char*, unsigned long) (CryptoImpl.cpp:142)
==306190==    by 0x51B9DFA: Aws::Utils::UUID::RandomUUID() (UUID.cpp:79)
==306190==    by 0x51E71A9: Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const (AWSClient.cpp:235)
==306190==    by 0x51EC1E6: Aws::Client::AWSJsonClient::MakeRequest(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const (AWSClient.cpp:910)
==306190==    by 0x4C2DD23: Aws::DynamoDB::DynamoDBClient::BatchWriteItem(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const (DynamoDBClient.cpp:292)
==306190==    by 0x4C2DDAA: Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}::operator()() const (DynamoDBClient.cpp:297)
==306190==    by 0x4CF135E: Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> std::__invoke_impl<Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError>, Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}&>(std::__invoke_other, Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}&) (invoke.h:60)
==306190==    by 0x4CE9E7E: std::enable_if<is_invocable_r_v<Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError>, Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}&>, std::enable_if>::type std::__invoke_r<Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError>, Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}&>(Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError>&&, (Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}&)...) (invoke.h:115)
==306190==    by 0x4CE35D9: std::__future_base::_Task_state<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}, std::allocator<int>, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> ()>::_M_run()::{lambda()#1}::operator()() const (future:1425)
==306190==    by 0x4D04A53: std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}, std::allocator<int>, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> ()>::_M_run()::{lambda()#1}, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >::operator()() const (future:1342)
==306190==    by 0x4D0005F: std::unique_ptr<std::__future_base::_Result<Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >, std::__future_base::_Result_base::_Deleter> std::__invoke_impl<std::unique_ptr<std::__future_base::_Result<Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}, std::allocator<int>, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> ()>::_M_run()::{lambda()#1}, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >&>(std::__invoke_other, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}, std::allocator<int>, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> ()>::_M_run()::{lambda()#1}, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >&) (invoke.h:60)
==306190==    by 0x4CFA8F7: std::enable_if<is_invocable_r_v<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}, std::allocator<int>, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> ()>::_M_run()::{lambda()#1}, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >&>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}, std::allocator<int>, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> ()>::_M_run()::{lambda()#1}, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >&>::type std::__invoke_r<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}, std::allocator<int>, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> ()>::_M_run()::{lambda()#1}, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >&>(std::enable_if&&, (std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>&&)...) (invoke.h:113)
==306190==    by 0x4CF13E0: std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> >, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}, std::allocator<int>, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> ()>::_M_run()::{lambda()#1}, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> > >::_M_invoke(std::_Any_data const&) (std_function.h:292)
==306190==    by 0x4D1233E: std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const (std_function.h:622)
==306190==    by 0x4D05DBC: std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) (future:564)
==306190==    by 0x4D31F52: void std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (invoke.h:73)
==306190==    by 0x4D2355C: std::__invoke_result<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>::type std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (invoke.h:95)
==306190==    by 0x4D11FDB: void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}::operator()() const (mutex:717)
==306190==    by 0x4D1200E: void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::operator()() const (mutex:722)
==306190==    by 0x4D1201F: void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::_FUN() (mutex:722)
==306190==    by 0x5850996: __pthread_once_slow (pthread_once.c:116)
==306190==    by 0x4D04F9A: __gthread_once(int*, void (*)()) (gthr-default.h:700)
==306190==    by 0x4D120C9: void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (mutex:729)
==306190==    by 0x4D05A28: std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) (future:404)
==306190==    by 0x4CE363B: std::__future_base::_Task_state<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#1}, std::allocator<int>, Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> ()>::_M_run() (future:1427)
==306190==    by 0x4D144BC: std::packaged_task<Aws::Utils::Outcome<Aws::DynamoDB::Model::BatchWriteItemResult, Aws::DynamoDB::DynamoDBError> ()>::operator()() (future:1560)
==306190==    by 0x4C2DDF1: Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2}::operator()() const (DynamoDBClient.cpp:298)
==306190==    by 0x4CBA3BE: void std::__invoke_impl<void, Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2}&>(std::__invoke_other, Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2}&) (invoke.h:60)
==306190==    by 0x4CB87D4: std::__invoke_result<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2}&>::type std::__invoke<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2}&>(std::__invoke_result&&, (Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2}&)...) (invoke.h:95)
==306190==    by 0x4CB7025: void std::_Bind<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2} ()>::__call<void>(std::tuple<>&&, std::_Index_tuple<>) (functional:416)
==306190==    by 0x4CB38E3: void std::_Bind<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2} ()>::operator()<, void>() (functional:499)
==306190==    by 0x4CA8F7D: void std::__invoke_impl<void, std::_Bind<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2} ()>&>(std::__invoke_other, std::_Bind<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2} ()>&) (invoke.h:60)
==306190==    by 0x4C9E13F: std::enable_if<is_invocable_r_v<void, std::_Bind<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2} ()>&>, std::enable_if>::type std::__invoke_r<void, std::_Bind<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2} ()>&>(void&&, (std::_Bind<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2} ()>&)...) (invoke.h:110)
==306190==    by 0x4C9469B: std::_Function_handler<void (), std::_Bind<Aws::DynamoDB::DynamoDBClient::BatchWriteItemCallable(Aws::DynamoDB::Model::BatchWriteItemRequest const&) const::{lambda()#2} ()> >::_M_invoke(std::_Any_data const&) (std_function.h:291)
==306190==    by 0x51B4A19: std::function<void ()>::operator()() const (std_function.h:622)
==306190==    by 0x5203936: Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void ()>&&)::{lambda()#1}::operator()() const (Executor.cpp:18)
==306190==    by 0x52060D5: void std::__invoke_impl<void, Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void ()>&&)::{lambda()#1}>(std::__invoke_other, Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void ()>&&)::{lambda()#1}&&) (invoke.h:60)
==306190==    by 0x520607C: std::__invoke_result<Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void ()>&&)::{lambda()#1}>::type std::__invoke<Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void ()>&&)::{lambda()#1}>(std::__invoke_result&&, (Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void ()>&&)::{lambda()#1}&&)...) (invoke.h:95)
==306190==    by 0x5206029: void std::thread::_Invoker<std::tuple<Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void ()>&&)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (thread:264)
==306190==    by 0x5205FFD: std::thread::_Invoker<std::tuple<Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void ()>&&)::{lambda()#1}> >::operator()() (thread:271)
==306190==    by 0x5205FE1: std::thread::_State_impl<std::thread::_Invoker<std::tuple<Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void ()>&&)::{lambda()#1}> > >::_M_run() (thread:215)
==306190==    by 0x59BFBEF: ??? (in /usr/local/lib64/libstdc++.so.6.0.28)
==306190==    by 0x5848FA2: start_thread (pthread_create.c:486)
==306190==    by 0x5D564CE: clone (clone.S:95)
==306190== 
==306190== LEAK SUMMARY:
==306190==    definitely lost: 336 bytes in 2 blocks
==306190==    indirectly lost: 528 bytes in 2 blocks
==306190==      possibly lost: 0 bytes in 0 blocks
==306190==    still reachable: 5,656 bytes in 21 blocks
==306190==         suppressed: 0 bytes in 0 blocks
==306190== Reachable blocks (those to which a pointer was found) are not shown.
==306190== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==306190== 
==306190== For counts of detected and suppressed errors, rerun with: -v
==306190== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)

Oct 11 '21 18:10 Radioguy00

Hi @Radioguy00 , Thanks for pointing this out to us!

Oct 12 '21 17:10 KaibaLopez

any updates? I have same problem

Oct 29 '21 02:10 maxjiang153

The leaks are in fact pretty severe as it looks like these "*callable" functions create a new thread for each call and each call add more leak.

Oct 31 '21 20:10 Radioguy00

It seems that each new thread created which uses s2n does not call the s2n_cleanup function as specified in the s2n documentation. The issue can be fixed in a "clumsy" way by applying the following patch. This is clumsy because it only corrects the 2 functions that I am using and the call of s2n cleanup in this file looks inappropriately located.

diff --git a/aws-cpp-sdk-dynamodb/source/DynamoDBClient.cpp b/aws-cpp-sdk-dynamodb/source/DynamoDBClient.cpp
index 21960541f9..c11a83badc 100644
--- a/aws-cpp-sdk-dynamodb/source/DynamoDBClient.cpp
+++ b/aws-cpp-sdk-dynamodb/source/DynamoDBClient.cpp
@@ -3,6 +3,7 @@
  * SPDX-License-Identifier: Apache-2.0.
  */

+#include "/home/builds/aws-sdk-cpp/crt/aws-crt-cpp/crt/s2n/api/s2n.h"
 #include <aws/core/utils/Outcome.h>
 #include <aws/core/auth/AWSAuthSigner.h>
 #include <aws/core/client/CoreErrors.h>
@@ -295,7 +296,7 @@ BatchWriteItemOutcome DynamoDBClient::BatchWriteItem(const BatchWriteItemRequest
 BatchWriteItemOutcomeCallable DynamoDBClient::BatchWriteItemCallable(const BatchWriteItemRequest& request) const
 {
   auto task = Aws::MakeShared< std::packaged_task< BatchWriteItemOutcome() > >(ALLOCATION_TAG, [this, request](){ return this->BatchWriteItem(request); } );
-  auto packagedFunction = [task]() { (*task)(); };
+  auto packagedFunction = [task]() { (*task)();s2n_cleanup(); };
   m_executor->Submit(packagedFunction);
   return task->get_future();
 }
@@ -1684,7 +1685,7 @@ QueryOutcome DynamoDBClient::Query(const QueryRequest& request) const
 QueryOutcomeCallable DynamoDBClient::QueryCallable(const QueryRequest& request) const
 {
   auto task = Aws::MakeShared< std::packaged_task< QueryOutcome() > >(ALLOCATION_TAG, [this, request](){ return this->Query(request); } );
-  auto packagedFunction = [task]() { (*task)(); };
+  auto packagedFunction = [task]() { (*task)(); s2n_cleanup();};
   m_executor->Submit(packagedFunction);
   return task->get_future();
 }

Nov 27 '21 22:11 Radioguy00

Any update on when this issue will be fixed?

Jul 12 '22 10:07 rahul-gundecha

I am seeing similar memory leaks.

However I am not using DynamoDB but a simple S3 client: ASan report:

=================================================================
==4395==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 504 byte(s) in 3 object(s) allocated from:
    #0 0x7f2c59e3e638 in __interceptor_malloc (/lib64/libasan.so.5+0x10c638)
    #1 0x7f2c5921c2d0 in CRYPTO_zalloc crypto/mem.c:230
    #2 0x7f2c58ea5482 in s2n_drbg_instantiate /root/.conan/data/s2n/1.3.9/exalead/testing/build/80d2f81d00bb87b9ba5e8d5ea7659d7f81457542/source_subfolder/crypto/s2n_drbg.c:155
    #3 0x7f2c58e9d0cd in s2n_defend_if_forked /root/.conan/data/s2n/1.3.9/exalead/testing/build/80d2f81d00bb87b9ba5e8d5ea7659d7f81457542/source_subfolder/utils/s2n_random.c:148
    #4 0x7f2c58e9d0cd in s2n_get_private_random_data /root/.conan/data/s2n/1.3.9/exalead/testing/build/80d2f81d00bb87b9ba5e8d5ea7659d7f81457542/source_subfolder/utils/s2n_random.c:179
    #5 0x7f2c58e9d2f7 in s2n_openssl_compat_rand /root/.conan/data/s2n/1.3.9/exalead/testing/build/80d2f81d00bb87b9ba5e8d5ea7659d7f81457542/source_subfolder/utils/s2n_random.c:301
    #6 0x7f2c58bda0a2 in Aws::Utils::Crypto::SecureRandomBytes_OpenSSLImpl::GetBytes(unsigned char*, unsigned long) /root/.conan/data/aws-sdk-cpp/1.9.100/exalead/testing/build/9f8b51551bacc92549dc0ee9b5774d6ad155701d/source_subfolder/aws-cpp-sdk-core/source/utils/crypto/openssl/CryptoImpl.cpp:142
    #7 0x7f2c58ce9762 in Aws::Utils::UUID::RandomUUID() /root/.conan/data/aws-sdk-cpp/1.9.100/exalead/testing/build/9f8b51551bacc92549dc0ee9b5774d6ad155701d/source_subfolder/aws-cpp-sdk-core/source/utils/UUID.cpp:79
    #8 0x7f2c58c41506 in Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&amp;, Aws::AmazonWebServiceRequest const&amp;, Aws::Http::HttpMethod, char const*, char const*, char const*) const /root/.conan/data/aws-sdk-cpp/1.9.100/exalead/testing/build/9f8b51551bacc92549dc0ee9b5774d6ad155701d/source_subfolder/aws-cpp-sdk-core/source/client/AWSClient.cpp:235
    #9 0x7f2c58c48c89 in Aws::Client::AWSXMLClient::MakeRequest(Aws::Http::URI const&amp;, Aws::AmazonWebServiceRequest const&amp;, Aws::Http::HttpMethod, char const*, char const*, char const*) const /root/.conan/data/aws-sdk-cpp/1.9.100/exalead/testing/build/9f8b51551bacc92549dc0ee9b5774d6ad155701d/source_subfolder/aws-cpp-sdk-core/source/client/AWSClient.cpp:1043
    #10 0x7f2c58a392e9 in Aws::S3::S3Client::HeadObject(Aws::S3::Model::HeadObjectRequest const&amp;) const /root/.conan/data/aws-sdk-cpp/1.9.100/exalead/testing/build/9f8b51551bacc92549dc0ee9b5774d6ad155701d/source_subfolder/aws-cpp-sdk-s3/source/S3Client.cpp:2215
    #11 0x7f2c587ccd77 in operator() /root/.conan/data/aws-sdk-cpp/1.9.100/exalead/testing/build/9f8b51551bacc92549dc0ee9b5774d6ad155701d/source_subfolder/aws-cpp-sdk-s3/source/S3Client.cpp:2220
    #12 0x7f2c587ccd77 in __invoke_impl&lt;Aws::Utils::Outcome&lt;Aws::S3::Model::HeadObjectResult, Aws::S3::S3Error&gt;, Aws::S3::S3Client::HeadObjectCallable(const Aws::S3::Model::HeadObjectRequest&amp;) const::&lt;lambda()&gt;&amp;&gt; /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/invoke.h:60
    #13 0x7f2c587ccd77 in __invoke&lt;Aws::S3::S3Client::HeadObjectCallable(const Aws::S3::Model::HeadObjectRequest&amp;) const::&lt;lambda()&gt;&amp;&gt; /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/invoke.h:96
    #14 0x7f2c587ccd77 in operator() /opt/rh/devtoolset-8/root/usr/include/c++/8/future:1421
    #15 0x7f2c587ccd77 in operator() /opt/rh/devtoolset-8/root/usr/include/c++/8/future:1339
    #16 0x7f2c587cde6b in _M_invoke /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:283
    #17 0x7f2c58a4b4e8 in std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;::operator()() const /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
    #18 0x7f2c58a4b4e8 in std::__future_base::_State_baseV2::_M_do_set(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*) /opt/rh/devtoolset-8/root/usr/include/c++/8/future:561
    #19 0x7f2c58a4c789 in void std::__invoke_impl&lt;void, void (std::__future_base::_State_baseV2::*)(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*), std::__future_base::_State_baseV2*, std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*&gt;(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&amp;&amp;)(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*), std::__future_base::_State_baseV2*&amp;&amp;, std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*&amp;&amp;, bool*&amp;&amp;) /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/invoke.h:73
    #20 0x7f2c58a4c789 in std::__invoke_result&lt;void (std::__future_base::_State_baseV2::*)(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*), std::__future_base::_State_baseV2*, std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*&gt;::type std::__invoke&lt;void (std::__future_base::_State_baseV2::*)(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*), std::__future_base::_State_baseV2*, std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*&gt;(void (std::__future_base::_State_baseV2::*&amp;&amp;)(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*), std::__future_base::_State_baseV2*&amp;&amp;, std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*&amp;&amp;, bool*&amp;&amp;) /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/invoke.h:95
    #21 0x7f2c58a4c789 in std::call_once&lt;void (std::__future_base::_State_baseV2::*)(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*), std::__future_base::_State_baseV2*, std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*&gt;(std::once_flag&amp;, void (std::__future_base::_State_baseV2::*&amp;&amp;)(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*), std::__future_base::_State_baseV2*&amp;&amp;, std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*&amp;&amp;, bool*&amp;&amp;)::{lambda()#1}::operator()() const /opt/rh/devtoolset-8/root/usr/include/c++/8/mutex:672
    #22 0x7f2c58a4c789 in std::call_once&lt;void (std::__future_base::_State_baseV2::*)(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*), std::__future_base::_State_baseV2*, std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*&gt;(std::once_flag&amp;, void (std::__future_base::_State_baseV2::*&amp;&amp;)(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*), std::__future_base::_State_baseV2*&amp;&amp;, std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*&amp;&amp;, bool*&amp;&amp;)::{lambda()#2}::operator()() const /opt/rh/devtoolset-8/root/usr/include/c++/8/mutex:677
    #23 0x7f2c58a4c789 in std::call_once&lt;void (std::__future_base::_State_baseV2::*)(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*), std::__future_base::_State_baseV2*, std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*&gt;(std::once_flag&amp;, void (std::__future_base::_State_baseV2::*&amp;&amp;)(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*), std::__future_base::_State_baseV2*&amp;&amp;, std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*&amp;&amp;, bool*&amp;&amp;)::{lambda()#2}::_FUN() /opt/rh/devtoolset-8/root/usr/include/c++/8/mutex:677
    #24 0x7f2c4e71e20a in __pthread_once_slow (/lib64/libpthread.so.0+0x620a)

Direct leak of 504 byte(s) in 3 object(s) allocated from:
[... similar callstack ...]

Indirect leak of 792 byte(s) in 3 object(s) allocated from:
[... similar callstack ...]

Indirect leak of 792 byte(s) in 3 object(s) allocated from:
[... similar callstack ...]

SUMMARY: AddressSanitizer: 2592 byte(s) leaked in 12 allocation(s).

Oct 04 '22 14:10 jeremy-coulon

Also seeing this issue. And it's not just with the *Callable functions.

Jan 07 '23 23:01 prm-james-hill

We fixed a memory leak with openssl here. Can you update to the latest version of the sdk and see if you are still getting any memory leaks?

Mar 06 '23 20:03 jmklix

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.

Mar 09 '23 00:03 github-actions[bot]

Confirming that I'm no longer seeing the leaks on the newest SDK commit

Apr 30 '23 03:04 prm-james-hill

aws-sdk-cpp aws-sdk-cpp copied to clipboard

Memory leak in functions Aws::DynamoDB::DynamoDBClient::xxxxxCallable

aws-sdk-cpp
aws-sdk-cpp copied to clipboard