apisix
apisix copied to clipboard
bug: APISIX v2.7 doesn't support S3(Object Storage API) signature_v4
Issue description
[root@qizhendong-dev1 conf]# s3cmd ls --debug
DEBUG: s3cmd version 2.1.0
DEBUG: ConfigParser: Reading file '/root/.s3cfg'
DEBUG: ConfigParser: host_base->10.139.15.11:9080
DEBUG: ConfigParser: host_bucket->10.139.15.11:9080
DEBUG: ConfigParser: bucket_location->us-east-1
DEBUG: ConfigParser: use_https->False
DEBUG: ConfigParser: access_key->mi...8_chars...
DEBUG: ConfigParser: secret_key->mi...7_chars...n
DEBUG: ConfigParser: signature_v2->False
DEBUG: ConfigParser: signature_v4->True
DEBUG: Updating Config.Config cache_file ->
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config verbosity -> 10
DEBUG: Unicodising 'ls' using UTF-8
DEBUG: Command: ls
DEBUG: CreateRequest: resource[uri]=/
DEBUG: Using signature v4
DEBUG: get_hostname(None): 10.139.15.11:9080
DEBUG: canonical_headers = host:10.139.15.11:9080
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20210810T115030Z
DEBUG: Canonical Request:
GET
/
host:10.139.15.11:9080
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20210810T115030Z
host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
----------------------
DEBUG: signature-v4 headers: {'x-amz-content-sha256': u'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': u'AWS4-HMAC-SHA256 Credential=minioadmin/20210810/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=b8a8aca635d76b99419e39fb8255ae21a0e631e3049463da96d88f456de313ee', 'x-amz-date': '20210810T115030Z'}
DEBUG: Processing request, please wait...
DEBUG: get_hostname(None): 10.139.15.11:9080
DEBUG: ConnMan.get(): creating new connection: http://10.139.15.11:9080
DEBUG: non-proxied HTTPConnection(10.139.15.11, 9080)
DEBUG: format_uri(): /
DEBUG: Sending request method_string='GET', uri=u'/', headers={'x-amz-content-sha256': u'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': u'AWS4-HMAC-SHA256 Credential=minioadmin/20210810/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=b8a8aca635d76b99419e39fb8255ae21a0e631e3049463da96d88f456de313ee', 'x-amz-date': '20210810T115030Z'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (http://10.139.15.11:9080#1)
DEBUG: Response:
{'data': '<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><Resource>/</Resource><RequestId>1699EFA47E3E44B3</RequestId><HostId>551cc69a-f48f-476e-b9be-3813b0903749</HostId></Error>',
'headers': {'accept-ranges': 'bytes',
'connection': 'keep-alive',
'content-length': '334',
'content-security-policy': 'block-all-mixed-content',
'content-type': 'application/xml',
'date': 'Tue, 10 Aug 2021 11:50:30 GMT',
'server': 'ICLOUD',
'strict-transport-security': 'max-age=31536000; includeSubDomains',
'vary': 'Origin, Accept-Encoding',
'x-amz-request-id': '1699EFA47E3E44B3',
'x-content-type-options': 'nosniff',
'x-xss-protection': '1; mode=block'},
'reason': 'Forbidden',
'status': 403}
DEBUG: S3Error: 403 (Forbidden)
DEBUG: HttpHeader: content-length: 334
DEBUG: HttpHeader: x-xss-protection: 1; mode=block
DEBUG: HttpHeader: x-content-type-options: nosniff
DEBUG: HttpHeader: content-security-policy: block-all-mixed-content
DEBUG: HttpHeader: accept-ranges: bytes
DEBUG: HttpHeader: strict-transport-security: max-age=31536000; includeSubDomains
DEBUG: HttpHeader: vary: Origin, Accept-Encoding
DEBUG: HttpHeader: server: ICLOUD
DEBUG: HttpHeader: connection: keep-alive
DEBUG: HttpHeader: x-amz-request-id: 1699EFA47E3E44B3
DEBUG: HttpHeader: date: Tue, 10 Aug 2021 11:50:30 GMT
DEBUG: HttpHeader: content-type: application/xml
DEBUG: ErrorXML: Code: 'SignatureDoesNotMatch'
DEBUG: ErrorXML: Message: 'The request signature we calculated does not match the signature you provided. Check your key and signing method.'
DEBUG: ErrorXML: Resource: '/'
DEBUG: ErrorXML: RequestId: '1699EFA47E3E44B3'
DEBUG: ErrorXML: HostId: '551cc69a-f48f-476e-b9be-3813b0903749'
ERROR: S3 error: 403 (SignatureDoesNotMatch): The request signature we calculated does not match the signature you provided. Check your key and signing method.
Environment
Bug report without environment information will be ignored or closed.
- apisix version (cmd:
apisix version
): 2.7 - OS (cmd:
uname -a
): centos 7.3 - OpenResty / Nginx version (cmd:
nginx -V
oropenresty -V
):1.19.3.2 - etcd version, if have (cmd: run
curl http://127.0.0.1:9090/v1/server_info
to get the info from server-info API): 3.4.0 - apisix-dashboard version, if have: no
- luarocks version, if the issue is about installation (cmd:
luarocks --version
): 3.4.0
Minimal test code / Steps to reproduce the issue
- setup minio
- use apisix as proxy and minio ip:port as upstream
- using s3cmd to list bucket
- s3cmd config file
host_base = 10.139.15.11:9080
host_bucket = 10.139.15.11:9080
bucket_location = us-east-1
use_https = False
access_key = minioadmin
secret_key = minioadmin
signature_v2 = False # work
signature_v4 = True # doesn't work
Bug report without steps to reproduce will be ignored or closed.
What's the actual result? (including assertion message & call stack if applicable)
What's the expected result?
Where are the APISIX related configurations? Please also show them.
Where are the APISIX related configurations? Please also show them.
APISIX all configs are
curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"upstream": {
"nodes": {
"10.139.15.11:9001": 1
},
"type": "roundrobin"
},
"uri": "/*"
}'
relation issue https://github.com/apache/apisix/issues/4803
If remove apisix proxy, directly using the MinIO endpoints with signature V4 on s3cmd client which can work.
How does s3cmd calculate the signature V4? Is it calculated from the headers? If it is, we can dump the header sent by the client and the header sent to the upstream, then compare them.
How does s3cmd calculate the signature V4? Is it calculated from the headers? If it is, we can dump the header sent by the client and the header sent to the upstream, then compare them.
It is a way. I know littile about s3 signture v4. This blog you can refer https://blog.csdn.net/m0_37263637/article/details/79553560. I valitated that using Ceph as the Object storage upstrean failure too.
The the way, This issue is nothing to with mirror proxy plugin.
This issue is import for me, but not urgent.
Based on your description, I think that some of the parameters involved in the signature calculation were updated after the request went through the APISIX proxy.
Is it necessary to use APISIX for proxying when there is a strict signature check between the SDK and the Serve? I don't think so.
Based on your description, I think that some of the parameters involved in the signature calculation were updated after the request went through the APISIX proxy.
Is it necessary to use APISIX for proxying when there is a strict signature check between the SDK and the Serve? I don't think so.
Maybe you are right, but there are many signature methods on client ans server, if one method can work but another doesn't which is strange for APISIX. I think that Object storage is a very import field for APISIX, we should try to support.
Then we need to do
- understand the logic of S3 signature_v4
- apisix avoid updating the parameters involved in the signature
How does s3cmd calculate the signature V4? Is it calculated from the headers? If it is, we can dump the header sent by the client and the header sent to the upstream, then compare them.
It is a way. I know littile about s3 signture v4. This blog you can refer blog.csdn.net/m0_37263637/article/details/79553560. I valitated that using Ceph as the Object storage upstrean failure too.
I take a short read to the blog. The signature V4 takes Headers & QueryString & hash of the body. Could you compare them across the request from the client and the one from APISIX?
How does s3cmd calculate the signature V4? Is it calculated from the headers? If it is, we can dump the header sent by the client and the header sent to the upstream, then compare them.
It is a way. I know littile about s3 signture v4. This blog you can refer blog.csdn.net/m0_37263637/article/details/79553560. I valitated that using Ceph as the Object storage upstrean failure too.
I take a short read to the blog. The signature V4 takes Headers & QueryString & hash of the body. Could you compare them across the request from the client and the one from APISIX?
s3cmd put test s3://test --debug
- s3cmd debug (using apisix got 403)
[root@status ~]# s3cmd put test s3://test --debug
DEBUG: s3cmd version 2.1.0
DEBUG: ConfigParser: Reading file '/root/.s3cfg'
DEBUG: ConfigParser: host_base->10.139.15.11:9080
DEBUG: ConfigParser: host_bucket->10.139.15.11:9080
DEBUG: ConfigParser: bucket_location->us-east-1
DEBUG: ConfigParser: use_https->False
DEBUG: ConfigParser: access_key->mi...8_chars...
DEBUG: ConfigParser: secret_key->mi...7_chars...n
DEBUG: ConfigParser: signature_v4->True
DEBUG: Updating Config.Config cache_file ->
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config verbosity -> 10
DEBUG: Unicodising 'put' using UTF-8
DEBUG: Unicodising 'test' using UTF-8
DEBUG: Unicodising 's3://test' using UTF-8
DEBUG: Command: put
INFO: No cache file found, creating it.
DEBUG: DeUnicodising u'test' using UTF-8
INFO: Compiling list of local files...
DEBUG: DeUnicodising u'test' using UTF-8
DEBUG: Unicodising 'test' using UTF-8
DEBUG: DeUnicodising u'test' using UTF-8
DEBUG: DeUnicodising u'test' using UTF-8
DEBUG: Unicodising '' using UTF-8
DEBUG: DeUnicodising u'test' using UTF-8
DEBUG: Unicodising 'test' using UTF-8
DEBUG: DeUnicodising u'test' using UTF-8
DEBUG: DeUnicodising u'test' using UTF-8
DEBUG: Applying --exclude/--include
DEBUG: CHECK: test
DEBUG: PASS: u'test'
INFO: Running stat() and reading/calculating MD5 values on 1 files, this may take some time...
DEBUG: DeUnicodising u'test' using UTF-8
DEBUG: doing file I/O to read md5 of test
DEBUG: DeUnicodising u'test' using UTF-8
INFO: Summary: 1 local files to upload
DEBUG: String 'root' encoded to 'root'
DEBUG: String 'root' encoded to 'root'
DEBUG: attr_header: {'x-amz-meta-s3cmd-attrs': u'atime:1628824603/ctime:1628824592/gid:0/gname:root/md5:ba1f2511fc30423bdbb183fe33f3dd0f/mode:33184/mtime:1628824592/uid:0/uname:root'}
DEBUG: DeUnicodising u'test' using UTF-8
DEBUG: DeUnicodising u'test' using UTF-8
DEBUG: DeUnicodising u'test' using UTF-8
DEBUG: CreateRequest: resource[uri]=/test
upload: 'test' -> 's3://test/test' [1 of 1]
DEBUG: DeUnicodising u'test' using UTF-8
DEBUG: Using signature v4
DEBUG: get_hostname(test): 10.139.15.11:9080
DEBUG: canonical_headers = content-length:4
content-type:text/plain
host:10.139.15.11:9080
x-amz-content-sha256:181210f8f9c779c26da1d9b2075bde0127302ee0e3fca38c9a83f5b1dd8e5d3b
x-amz-date:20210813T031712Z
x-amz-meta-s3cmd-attrs:atime:1628824603/ctime:1628824592/gid:0/gname:root/md5:ba1f2511fc30423bdbb183fe33f3dd0f/mode:33184/mtime:1628824592/uid:0/uname:root
x-amz-storage-class:STANDARD
DEBUG: Canonical Request:
PUT
/test/test
content-length:4
content-type:text/plain
host:10.139.15.11:9080
x-amz-content-sha256:181210f8f9c779c26da1d9b2075bde0127302ee0e3fca38c9a83f5b1dd8e5d3b
x-amz-date:20210813T031712Z
x-amz-meta-s3cmd-attrs:atime:1628824603/ctime:1628824592/gid:0/gname:root/md5:ba1f2511fc30423bdbb183fe33f3dd0f/mode:33184/mtime:1628824592/uid:0/uname:root
x-amz-storage-class:STANDARD
content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-meta-s3cmd-attrs;x-amz-storage-class
181210f8f9c779c26da1d9b2075bde0127302ee0e3fca38c9a83f5b1dd8e5d3b
----------------------
DEBUG: signature-v4 headers: {'x-amz-content-sha256': u'181210f8f9c779c26da1d9b2075bde0127302ee0e3fca38c9a83f5b1dd8e5d3b', 'content-length': '4', 'x-amz-storage-class': 'STANDARD', 'x-amz-meta-s3cmd-attrs': u'atime:1628824603/ctime:1628824592/gid:0/gname:root/md5:ba1f2511fc30423bdbb183fe33f3dd0f/mode:33184/mtime:1628824592/uid:0/uname:root', 'x-amz-date': '20210813T031712Z', 'content-type': 'text/plain', 'Authorization': u'AWS4-HMAC-SHA256 Credential=minioadmin/20210813/us-east-1/s3/aws4_request,SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-meta-s3cmd-attrs;x-amz-storage-class,Signature=f082b9207f64976c1b5069138558edd75ea7a01125ae7cae4fb64d9f9cbc7530'}
DEBUG: get_hostname(test): 10.139.15.11:9080
DEBUG: ConnMan.get(): creating new connection: http://10.139.15.11:9080
DEBUG: non-proxied HTTPConnection(10.139.15.11, 9080)
DEBUG: format_uri(): /test/test
4 of 4 100% in 0s 507.55 B/sDEBUG: ConnMan.put(): connection put back to pool (http://10.139.15.11:9080#1)
DEBUG: Response:
{'data': '<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><Key>test</Key><BucketName>test</BucketName><Resource>/test/test</Resource><RequestId>169ABF5F9C0D87D7</RequestId><HostId>fb0e6c83-4ebc-4fef-a280-1d5761dbf4c2</HostId></Error>',
'headers': {'accept-ranges': 'bytes',
'connection': 'keep-alive',
'content-length': '387',
'content-security-policy': 'block-all-mixed-content',
'content-type': 'application/xml',
'date': 'Fri, 13 Aug 2021 03:17:12 GMT',
'server': 'ICLOUD',
'strict-transport-security': 'max-age=31536000; includeSubDomains',
'vary': 'Origin, Accept-Encoding',
'x-amz-request-id': '169ABF5F9C0D87D7',
'x-content-type-options': 'nosniff',
'x-xss-protection': '1; mode=block'},
'reason': 'Forbidden',
'size': 4,
'status': 403}
4 of 4 100% in 0s 294.68 B/s done
DEBUG: S3Error: 403 (Forbidden)
DEBUG: HttpHeader: content-length: 387
DEBUG: HttpHeader: x-xss-protection: 1; mode=block
DEBUG: HttpHeader: x-content-type-options: nosniff
DEBUG: HttpHeader: content-security-policy: block-all-mixed-content
DEBUG: HttpHeader: accept-ranges: bytes
DEBUG: HttpHeader: strict-transport-security: max-age=31536000; includeSubDomains
DEBUG: HttpHeader: vary: Origin, Accept-Encoding
DEBUG: HttpHeader: server: ICLOUD
DEBUG: HttpHeader: connection: keep-alive
DEBUG: HttpHeader: x-amz-request-id: 169ABF5F9C0D87D7
DEBUG: HttpHeader: date: Fri, 13 Aug 2021 03:17:12 GMT
DEBUG: HttpHeader: content-type: application/xml
DEBUG: ErrorXML: Code: 'SignatureDoesNotMatch'
DEBUG: ErrorXML: Message: 'The request signature we calculated does not match the signature you provided. Check your key and signing method.'
DEBUG: ErrorXML: Key: 'test'
DEBUG: ErrorXML: BucketName: 'test'
DEBUG: ErrorXML: Resource: '/test/test'
DEBUG: ErrorXML: RequestId: '169ABF5F9C0D87D7'
DEBUG: ErrorXML: HostId: 'fb0e6c83-4ebc-4fef-a280-1d5761dbf4c2'
ERROR: S3 error: 403 (SignatureDoesNotMatch): The request signature we calculated does not match the signature you provided. Check your key and signing method.
- without apisix
[root@status ~]# s3cmd put test s3://test --debug
DEBUG: s3cmd version 2.1.0
DEBUG: ConfigParser: Reading file '/root/.s3cfg'
DEBUG: ConfigParser: host_base->10.139.15.11:9001
DEBUG: ConfigParser: host_bucket->10.139.15.11:9001
DEBUG: ConfigParser: bucket_location->us-east-1
DEBUG: ConfigParser: use_https->False
DEBUG: ConfigParser: access_key->mi...8_chars...
DEBUG: ConfigParser: secret_key->mi...7_chars...n
DEBUG: ConfigParser: signature_v4->True
DEBUG: Updating Config.Config cache_file ->
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config verbosity -> 10
DEBUG: Unicodising 'put' using UTF-8
DEBUG: Unicodising 'test' using UTF-8
DEBUG: Unicodising 's3://test' using UTF-8
DEBUG: Command: put
INFO: No cache file found, creating it.
DEBUG: DeUnicodising u'test' using UTF-8
INFO: Compiling list of local files...
DEBUG: DeUnicodising u'test' using UTF-8
DEBUG: Unicodising 'test' using UTF-8
DEBUG: DeUnicodising u'test' using UTF-8
DEBUG: DeUnicodising u'test' using UTF-8
DEBUG: Unicodising '' using UTF-8
DEBUG: DeUnicodising u'test' using UTF-8
DEBUG: Unicodising 'test' using UTF-8
DEBUG: DeUnicodising u'test' using UTF-8
DEBUG: DeUnicodising u'test' using UTF-8
DEBUG: Applying --exclude/--include
DEBUG: CHECK: test
DEBUG: PASS: u'test'
INFO: Running stat() and reading/calculating MD5 values on 1 files, this may take some time...
DEBUG: DeUnicodising u'test' using UTF-8
DEBUG: doing file I/O to read md5 of test
DEBUG: DeUnicodising u'test' using UTF-8
INFO: Summary: 1 local files to upload
DEBUG: String 'root' encoded to 'root'
DEBUG: String 'root' encoded to 'root'
DEBUG: attr_header: {'x-amz-meta-s3cmd-attrs': u'atime:1628824603/ctime:1628824592/gid:0/gname:root/md5:ba1f2511fc30423bdbb183fe33f3dd0f/mode:33184/mtime:1628824592/uid:0/uname:root'}
DEBUG: DeUnicodising u'test' using UTF-8
DEBUG: DeUnicodising u'test' using UTF-8
DEBUG: DeUnicodising u'test' using UTF-8
DEBUG: CreateRequest: resource[uri]=/test
upload: 'test' -> 's3://test/test' [1 of 1]
DEBUG: DeUnicodising u'test' using UTF-8
DEBUG: Using signature v4
DEBUG: get_hostname(test): 10.139.15.11:9001
DEBUG: canonical_headers = content-length:4
content-type:text/plain
host:10.139.15.11:9001
x-amz-content-sha256:181210f8f9c779c26da1d9b2075bde0127302ee0e3fca38c9a83f5b1dd8e5d3b
x-amz-date:20210813T031909Z
x-amz-meta-s3cmd-attrs:atime:1628824603/ctime:1628824592/gid:0/gname:root/md5:ba1f2511fc30423bdbb183fe33f3dd0f/mode:33184/mtime:1628824592/uid:0/uname:root
x-amz-storage-class:STANDARD
DEBUG: Canonical Request:
PUT
/test/test
content-length:4
content-type:text/plain
host:10.139.15.11:9001
x-amz-content-sha256:181210f8f9c779c26da1d9b2075bde0127302ee0e3fca38c9a83f5b1dd8e5d3b
x-amz-date:20210813T031909Z
x-amz-meta-s3cmd-attrs:atime:1628824603/ctime:1628824592/gid:0/gname:root/md5:ba1f2511fc30423bdbb183fe33f3dd0f/mode:33184/mtime:1628824592/uid:0/uname:root
x-amz-storage-class:STANDARD
content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-meta-s3cmd-attrs;x-amz-storage-class
181210f8f9c779c26da1d9b2075bde0127302ee0e3fca38c9a83f5b1dd8e5d3b
----------------------
DEBUG: signature-v4 headers: {'x-amz-content-sha256': u'181210f8f9c779c26da1d9b2075bde0127302ee0e3fca38c9a83f5b1dd8e5d3b', 'content-length': '4', 'x-amz-storage-class': 'STANDARD', 'x-amz-meta-s3cmd-attrs': u'atime:1628824603/ctime:1628824592/gid:0/gname:root/md5:ba1f2511fc30423bdbb183fe33f3dd0f/mode:33184/mtime:1628824592/uid:0/uname:root', 'x-amz-date': '20210813T031909Z', 'content-type': 'text/plain', 'Authorization': u'AWS4-HMAC-SHA256 Credential=minioadmin/20210813/us-east-1/s3/aws4_request,SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-meta-s3cmd-attrs;x-amz-storage-class,Signature=eb67f0a06a891874031e6dee7373459de3a95e796aef0efad4d8faf6e0950929'}
DEBUG: get_hostname(test): 10.139.15.11:9001
DEBUG: ConnMan.get(): creating new connection: http://10.139.15.11:9001
DEBUG: non-proxied HTTPConnection(10.139.15.11, 9001)
DEBUG: format_uri(): /test/test
4 of 4 100% in 0s 507.36 B/sDEBUG: ConnMan.put(): connection put back to pool (http://10.139.15.11:9001#1)
DEBUG: Response:
{'data': '',
'headers': {'accept-ranges': 'bytes',
'content-length': '0',
'content-security-policy': 'block-all-mixed-content',
'date': 'Fri, 13 Aug 2021 03:19:09 GMT',
'etag': '"ba1f2511fc30423bdbb183fe33f3dd0f"',
'server': 'MinIO',
'strict-transport-security': 'max-age=31536000; includeSubDomains',
'vary': 'Origin, Accept-Encoding',
'x-amz-request-id': '169ABF7AD32DA540',
'x-content-type-options': 'nosniff',
'x-xss-protection': '1; mode=block'},
'reason': 'OK',
'size': 4,
'status': 200}
4 of 4 100% in 0s 306.30 B/s done
DEBUG: MD5 sums: computed=ba1f2511fc30423bdbb183fe33f3dd0f, received=ba1f2511fc30423bdbb183fe33f3dd0f
Also need to provide request from the APISIX.
This issue has been marked as stale due to 350 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.
This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.