global state cleanup

[levongh]

Hello, I am using the azure SDK for cpp in a multiprocessing context where in the main process as an initial step I am sending a couple of requests through azure SDK. and then In subprocesses I create a new blob client and use it in subprocesses. an in some random scenario the program crashes on m_cleanThread.join()

As I can see there is a m_cleanThread which takes care of g_curlConnectionPool cleanup task. Is there any common place where I can do pre_fork global state cleanup?

Setup (please complete the following information if applicable):

• OS: [MacOS, Linux]
• Version of the Library used azure-core_1.11.1 azure-identity_1.6.0 azure-storage-blobs_12.10.0 azure-storage-common_12.5.0

[github-actions[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @EmmaZhu @Jinming-Hu @vinjiang.

[ahsonkhan]

Can you share a minimal, reproducible example (MRE) of an application which showcases the error you are hitting. That would be helpful in investigating the issue and provide guidance to your question.

[levongh]

@ahsonkhan Hoping it is enough to share as this is the core logic that is done. Note: I do not have any internal global states

below is provided the code where I create an AzureReader which is holding an internal client to send requests

#include <azure/core/http/http_status_code.hpp>
#include <azure/core/http/curl_transport.hpp>
#include <azure/storage/blobs.hpp>
#include <azure/identity.hpp>


std::unique_ptr<Azure::Storage::Blobs::BlobServiceClient> create_client(const std::string& blob_endpoint_)
{
    ::Azure::Storage::Blobs::BlobClientOptions client_options_
    Azure::Core::Http::CurlTransportOptions transport_opts;
    transport_opts.CAInfo = "/etc/ssl/cert.pem"

    transport_opts.SslVerifyPeer = true;

    std::make_shared<Azure::Core::Http::CurlTransport>(transport_opts);

    client_options_.Transport.Transport = std::make_shared<Azure::Core::Http::CurlTransport>(transport_opts);

    return std::make_unique<::Azure::Storage::Blobs::BlobServiceClient>(
        blob_endpoint_,
        client_options_);
}



class AzureReader
{
public:
    AzureReader(const std::string& container_name, const std::string& blob_endpoint_)
        : container_name_(container_name)
        , client_(create_client(create_client))
    {
    }

    std::vector<uint8_t> download(const std::string& path, std::pair<int64_t, int64_t> range)
    {
        ::Azure::Storage::Blobs::DownloadBlobOptions options;
        options.Range.Value().Offset = static_cast<int64_t>(range.first);
        options.Range.Value().Length = static_cast<int64_t>(range.second - range.first);
        client_->GetBlobContainerClient(config_.container_name_).Download(options).Value;
        
        auto length_returned = result.BodyStream->Length();
        std::vector<uint8_t> buffer(length_returned);
        result.BodyStream->ReadToCount(buffer.data(), length_returned);
        return buffer;
    }

private:
    std::string container_name_;
    std::unique_ptr<Azure::Storage::Blobs::BlobContainerClient> client_;
};

Also, I do have a pybind11 module where I expose AzureReader creation functionality and I just run random requests to container files

below is the Python side function calls

import mymodule
import multiprocessing
# multiprocessing.set_start_method("spawn", force=True)

mymodule.something()

def iterate(inp):
    mymodule.something()


if __name__ == "__main__":
    import os
    pool = multiprocessing.Pool(processes=1)

    a = list(zip([1]))

    pool.map_async(iterate, a)

    pool.close()
    pool.join()

[levongh]

Hi, pinging here to make sure the issue will not be closed by stale status.