GetObjectAsync with Aws::FStream as a ResponseStreamFactory creates the file locally and populates it with <Error><Code>NoSuchKey</Code> when the file doesn't exist on S3

[chreniuc]

Describe the bug

Hi,

we are trying to download a file from S3 using GetObjectAsync with Aws::FStream as a ResponseStreamFactory so it downloads it directly in a file locally.

The thing is that when that file doesn't exists on S3, it generates a local file with some xml content that says that the file cannot be found and it also logs this:

Encountered error while trying to download file. Error: HTTP response code: 404
Resolved remote host IP address: 52.218.56.235
Request ID: 
Exception name: 
Error message: 
6 response headers:
content-type : application/xml
date : Tue, 13 Sep 2022 12:19:40 GMT
server : AmazonS3
transfer-encoding : chunked
x-amz-id-2 : NlMtAG1blocEMIDJ6aobXMwEUVGPBKNlojHBgfno37fVgSfTe23ozBIjq1T9jRCtp7YsT2uHzc0=
x-amz-request-id : BJ7HZHSN2TB8F2FA.

Expected Behavior

The local file shouldn't be created locally and populated when it doesn't exist on S3.

Current Behavior

The file is created and populated with the following xml:

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>filename</Key><RequestId>DSC3VMT54C5BSWT5</RequestId><HostId>tIOhVB6b689IrIgqn/iuT4o2ciWKa3Of8P8fPLFUlDkX7r//LRgiMnsz1S4N1sKU5bJ5O2yu/o8=</HostId></Error>

Reproduction Steps

Code used to download the file and the handler:

Aws::S3::Model::GetObjectRequest request;
request.WithKey(key.c_str()).WithBucket(m_bucketName.c_str());

request.SetResponseStreamFactory([=]()
{
    return Aws::New<Aws::FStream>("S3DOWNLOAD", localFilePath,
        std::ios_base::out | std::ios_base::binary);
});

std::shared_ptr<Aws::Client::AsyncCallerContext> context =
    Aws::MakeShared<AsyncS3StorageCallerContext>("", shared_from_this());
context->SetUUID(key.c_str());

m_S3Client->GetObjectAsync(request, handler, context);

And our handler is this:

if (!outcome.IsSuccess())
    {
        LOG_ERROR("Encountered error while trying to download file. Error: %s.")
            % outcome.GetError();
        return;
    }

Possible Solution

Do not create the file locally if 404 is received.

Additional Information/Context

No response

AWS CPP SDK version used

1.8.1

Compiler and Version used

gcc 7.4

Operating System and version

Ubuntu 20.04

[jzakrzewski]

We had the same problem. It turns out the SDK has no idea about the file, because it only cares about the stream and the stream must be opened before the transfer. The file gets created by the FStream constructor.

Furthermore, if you wish to actually have an error message, you need to make it possible to rewind the stream. We ended up with std::ios_base::in | std::ios_base::out | std::ios_base::trunc | std::ios_base::binary flags for the stream, and we delete the file ourselves.