s3proxy icon indicating copy to clipboard operation
s3proxy copied to clipboard

Published 20 hours ago • verified publisher icon gaul

Failure to write into aws s3 bucket

Open interstellarPotato opened this issue 3 years ago • 7 comments

When attempting to put an object into aws s3 server rejects the request.

Curl command:

~ % curl -i -v -X PUT -T "/tmp/secret.txt" \
  http://127.0.0.1:8080/reducted-bucker-name/secret.txt

*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> PUT /reducted-bucket-name/secret.txt HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Length: 19
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
HTTP/1.1 100 Continue

* We are completely uploaded and fine
< HTTP/1.1 500 Server Error
HTTP/1.1 500 Server Error
< x-amz-request-id: 4442587FB7D0A2F9
x-amz-request-id: 4442587FB7D0A2F9
< Content-Length: 0
Content-Length: 0
< Server: Jetty(9.4.41.v20210516)
Server: Jetty(9.4.41.v20210516)

<
* Connection #0 to host 127.0.0.1 left intact
* Closing connection 0

Here is the stacks trace:

[s3proxy] D 09-10 03:05:27.175 S3Proxy-Jetty-14 o.g.s.S3ProxyHandlerJetty:88 |::] HttpResponseException without HttpResponse:
org.jclouds.http.HttpResponseException: Server rejected operation connecting to PUT https://reducted-bucket-name.s3-us-east-2.amazonaws.com/secret.txt HTTP/1.1
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:120)
	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
	at com.sun.proxy.$Proxy48.putObject(Unknown Source)
	at org.jclouds.s3.blobstore.S3BlobStore.putBlob(S3BlobStore.java:269)
	at org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:85)
	at org.gaul.s3proxy.S3ProxyHandler.handlePutBlob(S3ProxyHandler.java:1966)
	at org.gaul.s3proxy.S3ProxyHandler.doHandle(S3ProxyHandler.java:754)
	at org.gaul.s3proxy.S3ProxyHandlerJetty.handle(S3ProxyHandlerJetty.java:77)
	at org.gaul.shaded.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.gaul.shaded.org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.gaul.shaded.org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
	at org.gaul.shaded.org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
	at org.gaul.shaded.org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
	at org.gaul.shaded.org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.gaul.shaded.org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.gaul.shaded.org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.gaul.shaded.org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.gaul.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
	at org.gaul.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ProtocolException: Server rejected operation
	at sun.net.www.protocol.http.HttpURLConnection.expect100Continue(HttpURLConnection.java:1269)
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1348)
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:259)
	at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:299)
	at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:175)
	at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:66)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:97)
	... 23 common frames omitted

These is the list of headers:

[s3proxy] D 09-10 03:05:26.180 S3Proxy-Jetty-14 jclouds.signature:56 |::] << AWS4-HMAC-SHA256
[s3proxy] D 09-10 03:05:26.205 S3Proxy-Jetty-14 o.j.h.i.JavaUrlHttpCommandExecutorService:56 |::] Sending request -1521368375: PUT https://reducted-bucket-name.s3-us-east-2.amazonaws.com/secret.txt HTTP/1.1
[s3proxy] D 09-10 03:05:26.218 S3Proxy-Jetty-14 jclouds.wire:56 |::] >> "13;chunk-signature=a952045d41d1f5f65d5a5a93f00c48d479a45f388f0db13b271643e8863ee17a[\r][\n]"
[s3proxy] D 09-10 03:05:26.219 S3Proxy-Jetty-14 jclouds.wire:56 |::] >> "Hello secure world[\n]"
[s3proxy] D 09-10 03:05:26.220 S3Proxy-Jetty-14 jclouds.wire:56 |::] >> "[\r][\n]"
[s3proxy] D 09-10 03:05:26.221 S3Proxy-Jetty-14 jclouds.wire:56 |::] >> "0;chunk-signature=2b349cc174fc39017c4a618145bc9c55d25c4659b9872344da994b9e4dc2b340[\r][\n]"
[s3proxy] D 09-10 03:05:26.222 S3Proxy-Jetty-14 jclouds.wire:56 |::] >> "[\r][\n]"
[s3proxy] D 09-10 03:05:26.224 S3Proxy-Jetty-14 jclouds.headers:56 |::] >> PUT https://reducted-bucket-name.s3-us-east-2.amazonaws.com/secret.txt HTTP/1.1
[s3proxy] D 09-10 03:05:26.224 S3Proxy-Jetty-14 jclouds.headers:56 |::] >> Expect: 100-continue
[s3proxy] D 09-10 03:05:26.225 S3Proxy-Jetty-14 jclouds.headers:56 |::] >> Content-Encoding: aws-chunked
[s3proxy] D 09-10 03:05:26.226 S3Proxy-Jetty-14 jclouds.headers:56 |::] >> x-amz-decoded-content-length: 19
[s3proxy] D 09-10 03:05:26.226 S3Proxy-Jetty-14 jclouds.headers:56 |::] >> Content-Length: 192
[s3proxy] D 09-10 03:05:26.227 S3Proxy-Jetty-14 jclouds.headers:56 |::] >> Content-Type: application/unknown
[s3proxy] D 09-10 03:05:26.228 S3Proxy-Jetty-14 jclouds.headers:56 |::] >> Host: reducted-bucket-name.s3-us-east-2.amazonaws.com
[s3proxy] D 09-10 03:05:26.229 S3Proxy-Jetty-14 jclouds.headers:56 |::] >> x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD
[s3proxy] D 09-10 03:05:26.230 S3Proxy-Jetty-14 jclouds.headers:56 |::] >> X-Amz-Date: 20210910T030526Z
[s3proxy] D 09-10 03:05:26.230 S3Proxy-Jetty-14 jclouds.headers:56 |::] >> Authorization: AWS4-HMAC-SHA256 Credential=’XXXXXXXXXXXXXXX’/20210910/us-east-2/s3/aws4_request, SignedHeaders=content-encoding;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=2f213ccba7496cc1bfa74a7b70c2385c68cfb37e94f7e66103b952c2d34c8e3c
[s3proxy] D 09-10 03:05:26.231 S3Proxy-Jetty-14 jclouds.headers:56 |::] >> Content-Type: application/unknown
[s3proxy] D 09-10 03:05:26.232 S3Proxy-Jetty-14 jclouds.headers:56 |::] >> Content-Length: 192
[s3proxy] D 09-10 03:05:27.166 S3Proxy-Jetty-14 o.j.h.i.JavaUrlHttpCommandExecutorService:56 |::] Caught a protocol exception on a 100-continue PUT request. Attempting to retry.

I am sure I am missing something obvious. Would appreciate communities advice on this one. :) Additionally PUTing object into the same bucket via s3 cli works fine.

interstellarPotato avatar Sep 10 '21 03:09 interstellarPotato

Have you found any solution?

thakkarnirav avatar Nov 15 '22 22:11 thakkarnirav

Any update on this?

cocox avatar Apr 26 '23 07:04 cocox

+1

shibd avatar May 06 '23 02:05 shibd

Hi guys, to me I have to ignore unknown headers (it's a setting to enable) to make it work.

alifirat avatar May 10 '23 11:05 alifirat

How do you do it?

cocox avatar May 10 '23 14:05 cocox

I'm using the docker image and I've added the following variable:

environement:
 - ...   
 - S3PROXY_IGNORE_UNKNOWN_HEADERS=true

Be careful, this setting have been set to false for security reason I think.

alifirat avatar May 10 '23 14:05 alifirat

Running into this issue as well when taking rke2 etcd snapshots. I have tried setting S3PROXY_IGNORE_UNKNOWN_HEADERS=true to no avail.

rke2 uses minio-go/v7.0.33.

I found my issue higher up in logs in the original 400 response.

[s3proxy] D 02-06 07:43:43.322 S3Proxy-Jetty-11 o.j.h.i.JavaUrlHttpCommandExecutorService:56 |::] Receiving response -1477397346: HTTP/1.1 400 The metadata specified is invalid. It has characters that are not permitted.
[s3proxy] D 02-06 07:43:43.322 S3Proxy-Jetty-11 jclouds.headers:56 |::] << HTTP/1.1 400 The metadata specified is invalid. It has characters that are not permitted.

Looking at the C# Identifier docs, is there a way for s3proxy to be configure to prefix our metadata with underscores?

atsai1220 avatar Feb 06 '24 07:02 atsai1220