Segmentation fault in Aws::S3Crt::S3CrtClient::UploadPartCallable

[Johnparke888]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• [ x] I've gone though Developer Guide and API reference
• [ x] I've searched for previous similar issues and didn't find any solution

Describe the bug A clear and concise description of what the bug is. Attempting to do a multipart upload using the new S3Crt client. I started with a working version of code using the S3 client. When the code executes Aws::S3Crt::S3CrtClient::UploadPartCallable, I get a segmentation fault. Address sanitizer indicates an attempt to read from memory address which points to the zero page: ==11503==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x0000005888f3 bp 0x60300023b7b0 sp 0x7fff12af8210 T0) ==11503==The signal is caused by a READ memory access. ==11503==Hint: address points to the zero page. #0 0x5888f3 in Aws::S3Crt::S3CrtClient::UploadPartCallable(Aws::S3Crt::Model::UploadPartRequest const&) const (.../Linux/test2+0x5888f3) #1 0x4ddf5b in main /.../aws-multipart-upload-new/main.cpp:176 #2 0x7feaa9037544 in __libc_start_main (/lib64/libc.so.6+0x22544) #3 0x410241 (.../Linux/test2+0x410241)

I would be very grateful for any help.

John SDK version number 1.9.23 Platform/OS/Hardware/Device What are you running the sdk on? Red Hat Enterprise Linux Server release 7.5 (Maipo) gcc version 11.1.0 (GCC)

To Reproduce (observed behavior) Steps to reproduce the behavior (please share code)

int main () {

int i = 0; int j = 0; int position1 = 0; int position2 = 0;

int partNumber = 0; int outputBufferLength = 1024 * 1024 * 20; unsigned long int contentLength = 0;

const unsigned long int fixedPartSize = 1024 * 1024 * 8; int bytesRead = 0; const double throughput_target_gbps = 5;

Aws::Utils::Logging::InitializeAWSLogging( Aws::MakeSharedAws::Utils::Logging::DefaultLogSystem( "RunUnitTests", Aws::Utils::Logging::LogLevel::Trace, "aws_sdk_"));

std::string inputFile = "/brixhome/brixton/pdmbuild/jparkeTest/books/43254.txt";

std::ifstream infile(inputFile.c_str(), std::ios::binary); char outputBuffer = (char) malloc(outputBufferLength);

const Aws::String bucket_name = "test-cse-0cc24cfe-"; const Aws::String object_name = "test-file2.txt"; const std::string object_content = "text content."; const Aws::String region = "us-east-1"; std::string bucket = bucket_name; std::string key = object_name;

Aws::SDKOptions options; Aws::InitAPI(options); {

  Aws::S3Crt::ClientConfiguration clientConfiguration;
  clientConfiguration.region = region;
  clientConfiguration.throughputTargetGbps = throughput_target_gbps;
  clientConfiguration.partSize = fixedPartSize;
  Aws::S3Crt::S3CrtClient s3_client;

  Aws::S3Crt::Model::CreateMultipartUploadRequest multipartUploadRequest;
  multipartUploadRequest.SetBucket(bucket.c_str());
  multipartUploadRequest.SetKey(key.c_str());

  auto multipartUploadOutcome = s3_client.CreateMultipartUpload(multipartUploadRequest);
  std::string upload_id = multipartUploadOutcome.GetResult().GetUploadId();

  std::cout << "multipart upload id is:" << upload_id << std::endl;

  Aws::S3Crt::Model::UploadPartRequest uploadPartRequest;
  uploadPartRequest.SetBucket(bucket.c_str());
  uploadPartRequest.SetKey(key.c_str());

  auto stream_ptr =
     Aws::MakeShared<Aws::StringStream>("PutObjectInputStream",
     std::stringstream::in | std::stringstream::out | std::stringstream::binary);

  Aws::S3Crt::Model::CompleteMultipartUploadRequest completeMultipartUploadRequest;
  Aws::S3Crt::Model::CompletedMultipartUpload completedMultipartUpload;

  //**************************
  // Set Content Length
  //**************************    

  infile.read(outputBuffer, fixedPartSize);
  bytesRead = infile.gcount();
  std::cout << "read(" << bytesRead << ")" << std::endl;

  stream_ptr->write(reinterpret_cast<char*> (outputBuffer), bytesRead);

  uploadPartRequest.SetUploadId(upload_id.c_str());

  Aws::Utils::ByteBuffer part_md5(Aws::Utils::HashingUtils::CalculateMD5(*stream_ptr));
  uploadPartRequest.SetContentMD5(Aws::Utils::HashingUtils::Base64Encode(part_md5));

  uploadPartRequest.SetBody(stream_ptr);

  auto start_pos = stream_ptr->tellg();

  stream_ptr->seekg(0LL, stream_ptr->end);

  partNumber ++;
  contentLength = static_cast<long> (stream_ptr->tellg());

  std::cout << "tellg (" << static_cast<long> (stream_ptr->tellg()) << ")" << std::endl;
  std::cout << "start position (" << start_pos << ")" << std::endl;
  std::cout << "part Number (" << partNumber << ")" << std::endl;
  std::cout << "content length set to(" << contentLength << ")" << std::endl;

  uploadPartRequest.SetContentLength(contentLength);
  uploadPartRequest.SetPartNumber(partNumber);
  stream_ptr->seekg(start_pos);

  std::cout << "upload id(" <<  uploadPartRequest.GetUploadId () << ")" << std::endl;

SEGV ===> auto uploadPartOutcomeCallable = s3_client.UploadPartCallable(uploadPartRequest);

  //**************************
  // finish upload
  //**************************

  completeMultipartUploadRequest.SetBucket(bucket.c_str());
  completeMultipartUploadRequest.SetKey(key.c_str());
  completeMultipartUploadRequest.SetUploadId(upload_id.c_str());

  Aws::S3Crt::Model::UploadPartOutcome uploadPartOutcome = uploadPartOutcomeCallable.get();
  Aws::S3Crt::Model::CompletedPart completedPart;

  completedPart.SetPartNumber(partNumber);
  Aws::S3Crt::Model::UploadPartResult uploadPartResult = uploadPartOutcome.GetResult();
  auto etag = uploadPartOutcome.GetResult().GetETag();
  Aws::S3Crt::Model::UploadPartOutcome response = s3_client.UploadPart(uploadPartRequest);

  completedPart.SetETag(etag);
  completedMultipartUpload.AddParts(completedPart);

  /*
   * Complete the Multipart Upload
   *             
   */
  completeMultipartUploadRequest.WithMultipartUpload(completedMultipartUpload);

  auto completeMultipartUploadOutcome = s3_client.CompleteMultipartUpload(completeMultipartUploadRequest);

  if (!completeMultipartUploadOutcome.IsSuccess())
  {
     auto error = completeMultipartUploadOutcome.GetError();
     std::cout << error << error.GetExceptionName() << ": " << error.GetMessage() << std::endl;

  }

} Aws::Utils::Logging::ShutdownAWSLogging(); Aws::ShutdownAPI(options);

Expected behavior A clear and concise description of what you expected to happen. File should upload. aws_sdk_2021-05-21-13.log asan.log.11503.log

Logs/output If applicable, add logs or error output.

Program output: multipart upload id is:fk95A9gWIUaO7KPwA462BgBBZjp8Bcq..Q4qxIO8IWbOZBkcsxQtQbWnQ_0c8zXQ0EeXWrqCQ8ESY3lS197dR4PT9.F2mdsHsmnVACv9Kxk32Y53Nmjfy8tlpDWgSNEI read(1178782) tellg (1178782) start position (0) part Number (1) content length set to(1178782) upload id(fk95A9gWIUaO7KPwA462BgBBZjp8Bcq..Q4qxIO8IWbOZBkcsxQtQbWnQ_0c8zXQ0EeXWrqCQ8ESY3lS197dR4PT9.F2mdsHsmnVACv9Kxk32Y53Nmjfy8tlpDWgSNEI) AddressSanitizer:DEADLYSIGNAL Aborted

To enable logging, set the following system properties:

REMEMBER TO SANITIZE YOUR PERSONAL INFO

options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Trace;
Aws::InitAPI(options)

Additional context Add any other context about the problem here. gdb output:

Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00000000004aa783 in Aws::S3Crt::S3CrtClient::UploadPartCallable(Aws::S3Crt::Model::UploadPartRequest const&) const () [Current thread is 1 (Thread 0x7f4cc995f800 (LWP 10892))] (gdb) where #0 0x00000000004aa783 in Aws::S3Crt::S3CrtClient::UploadPartCallable(Aws::S3Crt::Model::UploadPartRequest const&) const () #1 0x000000000040da87 in main () at main.cpp:176

[Johnparke888]

Additional information on code where crash occurs:

Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00000000004c17c0 in Aws::Utils::Threading::Executor::Submit<Aws::S3Crt::S3CrtClient::UploadPartCallable(const Aws::S3Crt::Model::UploadPartRequest&) const::<lambda()>&>(struct {...} &) (this=0x0, fn=...) at /usr/aws-sdk-cpp/aws-cpp-sdk-core/include/aws/core/utils/threading/Executor.h:42 42 return SubmitToThread(std::move(callable)); [Current thread is 1 (Thread 0x7fe61f705800 (LWP 30129))] (gdb) where #0 0x00000000004c17c0 in Aws::Utils::Threading::Executor::Submit<Aws::S3Crt::S3CrtClient::UploadPartCallable(const this value of 0x0 ==>Aws::S3Crt::Model::UploadPartRequest&) const::<lambda()>&>(struct {...} &) (this=0x0, fn=...) at /usr/aws-sdk-cpp/aws-cpp-sdk-core/include/aws/core/utils/threading/Executor.h:42 #1 0x0000000000444cfb in Aws::S3Crt::S3CrtClient::UploadPartCallable (this=0x7ffe6927e990, request=...) at /usr/aws-sdk-cpp/aws-cpp-sdk-s3-crt/source/S3CrtClient.cpp:4071 #2 0x000000000040cef7 in main () at main.cpp:176

4068 { 4069 auto task = Aws::MakeShared< std::packaged_task< UploadPartOutcome() > >(ALLOCATION_TAG, this, request{ return this->UploadPart(request); } ); 4070 auto packagedFunction = task { (*task)(); }; SEGV here ==>4071 m_executor->Submit(packagedFunction); 4072 return task->get_future(); 4073 }

           * Send function and its arguments to the SubmitToThread function.
             */
            template<class Fn, class ... Args>
            bool Submit(Fn&& fn, Args&& ... args)
            {
                std::function<void()> callable{ std::bind(std::forward<Fn>(fn), std::forward<Args>(args)...) };
          ==>      return SubmitToThread(std::move(callable));
            }

[KaibaLopez]

Hi @Johnparke888 , Alright, thanks for bringing this up to us. I'll see if I can figure out what's going on... But in the mean time, do you get the same error if you try using the transfer manager or the s3 client (instead of the s3crt) ?

[Johnparke888]

I get the same error if I use the S3 client instead of S3Crt. I don’t use the transfer manager.

I have dropped back to 1.8 in the meantime.

Appreciate your help.

John

From: Gilberto Lopez Zayas @.> Sent: Monday, May 24, 2021 1:33 PM To: aws/aws-sdk-cpp @.> Cc: Johnparke888 @.>; Mention @.> Subject: Re: [aws/aws-sdk-cpp] Segmentation fault in Aws::S3Crt::S3CrtClient::UploadPartCallable (#1654)

Hi @Johnparke888 https://github.com/Johnparke888 , Alright, thanks for bringing this up to us. I'll see if I can figure out what's going on... But in the mean time, do you get the same error if you try using the transfer manager or the s3 client (instead of the s3crt) ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/aws/aws-sdk-cpp/issues/1654#issuecomment-847214231 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ATX7IAFJE4L5ZSWXTPOYYK3TPKEWTANCNFSM45JJMJXA . https://github.com/notifications/beacon/ATX7IAAMCISP6ECO4M4USZTTPKEWTA5CNFSM45JJMJXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOGJ7XNFY.gif

[KaibaLopez]

@Johnparke888 , Wait so then the error doesn't happen on 1.8? With the same code? Also I should mention, when using the S3Crt client, it is recommended that you use only the Get and Upload functions, it will do multipart if needed on the background for you, dunno if this helps in your case since the s3 client still gives you the error, but yea thought I'd mention in case you want to try that.

[Johnparke888]

That’s right. The 1.8 code doesn’t get the segmentation fault.

I thought from reading the docs that, in 1.9, the Upload would do multipart if needed as you said.

Does it do this for a stream buffer, or does it have to be from a source file?

If I can do it from a stream than I could use 1.9.

Thanks again,

John

From: Gilberto Lopez Zayas @.> Sent: Monday, May 24, 2021 4:35 PM To: aws/aws-sdk-cpp @.> Cc: Johnparke888 @.>; Mention @.> Subject: Re: [aws/aws-sdk-cpp] Segmentation fault in Aws::S3Crt::S3CrtClient::UploadPartCallable (#1654)

@Johnparke888 https://github.com/Johnparke888 , Wait so then the error doesn't happen on 1.8? With the same code? Also I should mention, when using the S3Crt client, it is recommended that you use only the Get and Upload functions, it will do multipart if needed on the background for you, dunno if this helps in your case since the s3 client still gives you the error, but yea thought I'd mention in case you want to try that.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/aws/aws-sdk-cpp/issues/1654#issuecomment-847316991 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ATX7IABXXOZPTKJDEDZVMZ3TPK2A7ANCNFSM45JJMJXA . https://github.com/notifications/beacon/ATX7IACYVKEXFL6LRFB7NO3TPK2A7A5CNFSM45JJMJXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOGKAQP7Y.gif

[pablo-carbonell]

The problem has to do with the executor in ClientConfiguration being a nullptr. You must create an instance of executor before making any calls to any ...Callable method. Use default if you don't care too much: config.executor = std::make_shared<Aws::Utils::Threading::DefaultExecutor>(); The SDK doesn't initialize either of S3CrtClient::m_executor or ClientConfiguration::executor, so they stay empty std::shared_ptrs. Other than that, there are a couple things that could be done from the SDK's point of view, either provide a default when the S3CrtClient initializes, when Aws::S3Crt::ClientConfiguration initializes, or warn/check when the shared_ptris empty.

[grrtrr]

The problem with uninitialized/empty m_executor is resolved by #1884, as it initializes the base class first.

[jmklix]

This should be fixed now with the above PR. Please let me know if you are still getting this seg fault.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.