Missing ETag header when downloading files from s3proxy

[bthdimension]

When I do a streaming download using the aws adk, I get the following error: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: Response missing required ETag header.

The streaming is done as follows:

s3AsyncClient.getObject(
                r -> r.bucket(bucket).key(key),
                AsyncResponseTransformer.toBlockingInputStream());
InputStream inputStream = responseFuture.join();
IOUtils.copy(inputStream, fileOutputStream);

I'm running s3proxy within a JUnit test and set it up as follows:

        Properties properties = new Properties();
        properties.setProperty(S3ProxyConstants.PROPERTY_AUTHORIZATION, "aws-v2-or-v4");
        properties.setProperty(S3ProxyConstants.PROPERTY_IDENTITY, accessKeyId);
        properties.setProperty(S3ProxyConstants.PROPERTY_CREDENTIAL, secretAccessKey);
        properties.setProperty(S3ProxyConstants.PROPERTY_IGNORE_UNKNOWN_HEADERS, "true");
        properties.setProperty(S3ProxyConstants.PROPERTY_ENDPOINT, endpoint);
        properties.setProperty(FilesystemConstants.PROPERTY_BASEDIR, TestFileManagement.S3_BASE_DIRECTORY);

        context = ContextBuilder
                .newBuilder("filesystem")
                .credentials("identity", "credential")
                .overrides(properties)
                .build(BlobStoreContext.class);

        s3Proxy = S3Proxy.Builder.fromProperties(properties)
                .blobStore(context.getBlobStore())
                .build();

There are some randomly generated files in the S3_BASE_DIRECTORY.

My s3proxy version is 1.9.0 due to the issue mentioned here (see my last comment): https://github.com/gaul/s3proxy/issues/535 My aws sdk version is:

• software.amazon.awssdk:s3:2.24.3
• software.amazon.awssdk.crt:aws-crt:0.29.10

[gaul]

S3Proxy includes the ETag if it is exists. For the filesystem backend, this is only present if S3Proxy created the object which stores the tag in the extended attributes if they are supported (I believe all platforms with modern Java versions). If you are calling getBlob on an objected created outside S3Proxy, e.g., exposing an existing directory via S3, this attribute will not be present and thus S3Proxy will not include the header. I am curious about Response missing required ETag header. -- this was not always a requirement so perhaps there is some regression in the AWS library or configuration you need to use?

[bthdimension]

Thank you very much for the quick answer. If I use the S3 client to upload the files, they indeed have the ETag. Is there a way to set these attributes manually? Otherwise I'll just use this workaround. Thanks!

[gaul]

S3Proxy (via Apache jclouds) previously tried computing ETag on demand when it was not available in apache/jclouds@496e27f1afa32b90d0656c3f21387cf68a30cb31. This was reverted in apache/jclouds@86e947ddedfa1a13976f9debeadae1da6f2f0190 since it was computationally expensive for large objects, particularly when listing objects.

Could you share why your library is enforcing the presence of ETag? This should have a configuration knob to disable the behavior.

[bthdimension]

The libraries I'm using are 'software.amazon.awssdk:s3:2.24.3' and 'software.amazon.awssdk.crt:aws-crt:0.29.10'. The client that seems to be enforcing it is "software.amazon.awssdk.services.s3.S3AsyncClient" and it is specifically the CRT client.

I create the client as follows:

S3AsyncClient.crtBuilder()
                .endpointOverride(URI.create(url))
                .forcePathStyle(true)
                .credentialsProvider(StaticCredentialsProvider.create(credentials))
                .region(Region.US_EAST_1)
                .targetThroughputInGbps(targetThroughputInGbps)
                .minimumPartSizeInBytes(minimumPartSizeInBytes)
                .checksumValidationEnabled(false)
                .build()

When creating the CRT client, I' can't really find an option to disable the ETag check. I was only able to disable the checksum validation.

[zjl578254155]

.forcePathStyle(true)

Hello, may I ask if this issue has been resolved? I am creating the download file in this way, but there is still an error message saying that Etag is missing. I don't know where the problem lies?