s3fs-fuse icon indicating copy to clipboard operation
s3fs-fuse copied to clipboard

Published 20 hours ago • verified publisher icon s3fs-fuse

Use full bucket url

Open hMED22 opened this issue 4 years ago • 4 comments
trafficstars

Additional Information

Can you allow using the full bucket url and omitting the bucket name argument.

Version of s3fs being used (s3fs --version)

V1.89

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

2.9.9

s3fs command line used, if applicable

s3fs -o public_bucket=1 -o url=https://cse-cic-ids2018.s3.amazonaws.com/ cse-cic-ids2018 /mnt/cse-cic-ids2018

or

s3fs -o public_bucket=1 -o url=https://cse-cic-ids2018.s3.amazonaws.com/ /mnt/cse-cic-ids2018

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

# s3fs -o public_bucket=1 -o url=https://cse-cic-ids2018.s3.amazonaws.com/ cse-cic-ids2018 /mnt/cse-cic-ids2018 -f -o dbglevel=info -o
 curldbg
2021-10-10T11:59:58.343Z [CRT] s3fs_logger.cpp:LowSetLogLevel(219): change debug level from [CRT] to [INF] 
2021-10-10T11:59:58.344Z [INF]     s3fs.cpp:set_mountpoint_attribute(4019): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755)
2021-10-10T11:59:58.362Z [INF] curl.cpp:InitMimeType(436): Loaded mime information from /etc/mime.types
2021-10-10T11:59:58.363Z [INF] fdcache_stat.cpp:CheckCacheFileStatTopDir(79): The path to cache top dir is empty, thus not need to check permission.
2021-10-10T11:59:58.364Z [INF] s3fs.cpp:s3fs_init(3331): init v1.89(commit:8c58ba8) with OpenSSL
2021-10-10T11:59:58.364Z [INF] s3fs.cpp:s3fs_check_service(3447): check services.
2021-10-10T11:59:58.364Z [INF]       curl.cpp:CheckBucket(3364): check a bucket.
2021-10-10T11:59:58.364Z [INF]       curl_util.cpp:prepare_url(250): URL is https://cse-cic-ids2018.s3.amazonaws.com/cse-cic-ids2018/
2021-10-10T11:59:58.364Z [INF]       curl_util.cpp:prepare_url(283): URL changed is https://cse-cic-ids2018.cse-cic-ids2018.s3.amazonaws.com/
2021-10-10T11:59:58.364Z [INF]       curl.cpp:insertV4Headers(2640): computing signature [GET] [/] [] []
2021-10-10T11:59:58.364Z [INF]       curl_util.cpp:url_to_host(327): url is https://cse-cic-ids2018.s3.amazonaws.com
2021-10-10T11:59:58.388Z [CURL DBG] *   Trying 52.216.109.211:443...
2021-10-10T11:59:58.720Z [CURL DBG] * Connected to cse-cic-ids2018.cse-cic-ids2018.s3.amazonaws.com (52.216.109.211) port 443 (#0)
2021-10-10T11:59:58.772Z [CURL DBG] * successfully set certificate verify locations:
2021-10-10T11:59:58.772Z [CURL DBG] *  CAfile: /etc/ssl/certs/ca-certificates.crt
2021-10-10T11:59:58.772Z [CURL DBG] *  CApath: none
2021-10-10T11:59:58.773Z [CURL DBG] * TLSv1.3 (OUT), TLS handshake, Client hello (1):
2021-10-10T11:59:58.925Z [CURL DBG] * TLSv1.3 (IN), TLS handshake, Server hello (2):
2021-10-10T11:59:58.925Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Certificate (11):
2021-10-10T11:59:58.927Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
2021-10-10T11:59:58.928Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Server finished (14):
2021-10-10T11:59:58.929Z [CURL DBG] * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
2021-10-10T11:59:58.929Z [CURL DBG] * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
2021-10-10T11:59:58.929Z [CURL DBG] * TLSv1.2 (OUT), TLS handshake, Finished (20):
2021-10-10T11:59:59.130Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Finished (20):
2021-10-10T11:59:59.130Z [CURL DBG] * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
2021-10-10T11:59:59.130Z [CURL DBG] * Server certificate:
2021-10-10T11:59:59.130Z [CURL DBG] *  subject: C=US; ST=Washington; L=Seattle; O=Amazon.com, Inc.; CN=*.s3.amazonaws.com
2021-10-10T11:59:59.130Z [CURL DBG] *  start date: Jan 11 00:00:00 2021 GMT
2021-10-10T11:59:59.130Z [CURL DBG] *  expire date: Feb 11 23:59:59 2022 GMT
2021-10-10T11:59:59.131Z [CURL DBG] *  subjectAltName does not match cse-cic-ids2018.cse-cic-ids2018.s3.amazonaws.com
2021-10-10T11:59:59.131Z [CURL DBG] * SSL: no alternative certificate subject name matches target host name 'cse-cic-ids2018.cse-cic-ids2018.s3.amazonaws.com'
2021-10-10T11:59:59.131Z [CURL DBG] * Closing connection 0
2021-10-10T11:59:59.131Z [CURL DBG] * TLSv1.2 (OUT), TLS alert, close notify (256):
2021-10-10T11:59:59.136Z [ERR] curl.cpp:RequestPerform(2414): ### CURLE_SSL_CACERT
2021-10-10T11:59:59.137Z [INF] curl.cpp:RequestPerform(2474): ### retrying...
2021-10-10T11:59:59.137Z [INF]       curl.cpp:RemakeHandle(2078): Retry request. [type=5][url=https://cse-cic-ids2018.cse-cic-ids2018.s3.amazonaws.com/][path=/]
2021-10-10T11:59:59.137Z [INF]       curl.cpp:insertV4Headers(2640): computing signature [GET] [/] [] []
2021-10-10T11:59:59.137Z [INF]       curl_util.cpp:url_to_host(327): url is https://cse-cic-ids2018.s3.amazonaws.com
2021-10-10T11:59:59.137Z [CURL DBG] * Hostname cse-cic-ids2018.cse-cic-ids2018.s3.amazonaws.com was found in DNS cache
2021-10-10T11:59:59.137Z [CURL DBG] *   Trying 52.216.109.211:443...
2021-10-10T11:59:59.268Z [CURL DBG] * Connected to cse-cic-ids2018.cse-cic-ids2018.s3.amazonaws.com (52.216.109.211) port 443 (#1)
2021-10-10T11:59:59.314Z [CURL DBG] * successfully set certificate verify locations:
2021-10-10T11:59:59.314Z [CURL DBG] *  CAfile: /etc/ssl/certs/ca-certificates.crt
2021-10-10T11:59:59.314Z [CURL DBG] *  CApath: none
2021-10-10T11:59:59.314Z [CURL DBG] * TLSv1.3 (OUT), TLS handshake, Client hello (1):
2021-10-10T11:59:59.538Z [CURL DBG] * TLSv1.3 (IN), TLS handshake, Server hello (2):
2021-10-10T11:59:59.539Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Certificate (11):
2021-10-10T11:59:59.541Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
2021-10-10T11:59:59.541Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Server finished (14):
2021-10-10T11:59:59.542Z [CURL DBG] * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
2021-10-10T11:59:59.542Z [CURL DBG] * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
2021-10-10T11:59:59.543Z [CURL DBG] * TLSv1.2 (OUT), TLS handshake, Finished (20):
2021-10-10T11:59:59.847Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Finished (20):
2021-10-10T11:59:59.848Z [CURL DBG] * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
2021-10-10T11:59:59.848Z [CURL DBG] * Server certificate:
2021-10-10T11:59:59.848Z [CURL DBG] *  subject: C=US; ST=Washington; L=Seattle; O=Amazon.com, Inc.; CN=*.s3.amazonaws.com
2021-10-10T11:59:59.848Z [CURL DBG] *  start date: Jan 11 00:00:00 2021 GMT
2021-10-10T11:59:59.848Z [CURL DBG] *  expire date: Feb 11 23:59:59 2022 GMT
2021-10-10T11:59:59.848Z [CURL DBG] *  subjectAltName does not match cse-cic-ids2018.cse-cic-ids2018.s3.amazonaws.com
2021-10-10T11:59:59.848Z [CURL DBG] * SSL: no alternative certificate subject name matches target host name 'cse-cic-ids2018.cse-cic-ids2018.s3.amazonaws.com'
2021-10-10T11:59:59.848Z [CURL DBG] * Closing connection 1
2021-10-10T11:59:59.848Z [CURL DBG] * TLSv1.2 (OUT), TLS alert, close notify (256):
2021-10-10T11:59:59.853Z [ERR] curl.cpp:RequestPerform(2414): ### CURLE_SSL_CACERT
2021-10-10T11:59:59.853Z [ERR] curl.cpp:RequestPerform(2425): curlCode: 60  msg: SSL peer certificate or SSH remote key was not OK
2021-10-10T11:59:59.853Z [ERR] curl.cpp:CheckBucket(3396): Check bucket failed, S3 response: 
2021-10-10T11:59:59.853Z [CRT] s3fs.cpp:s3fs_check_service(3524): unable to connect(host=https://cse-cic-ids2018.s3.amazonaws.com) - result of checking service.
2021-10-10T11:59:59.853Z [INF] curl_handlerpool.cpp:ReturnHandler(110): Pool full: destroy the oldest handler
2021-10-10T11:59:59.853Z [ERR] s3fs.cpp:s3fs_exit_fuseloop(3321): Exiting FUSE event loop due to errors

2021-10-10T11:59:59.855Z [INF] s3fs.cpp:s3fs_destroy(3389): destroy


# s3fs -o public_bucket=1 -o url=https://cse-cic-ids2018.s3.amazonaws.com/ /mnt/cse-cic-ids2018 -o dbglevel=info -f -o curldbg
2021-10-10T12:02:58.233Z [CRT] s3fs_logger.cpp:LowSetLogLevel(219): change debug level from [CRT] to [INF] 
2021-10-10T12:02:58.241Z [INF] curl.cpp:InitMimeType(436): Loaded mime information from /etc/mime.types
s3fs: BUCKET /mnt/cse-cic-ids2018 -- bucket name contains an illegal character.

hMED22 avatar Oct 10 '21 12:10 hMED22

@hMED22 I'm sorry for the late reply.

The content of your log seems to be the same as #1945. This issue was posted earlier than that, but please refer to that comment

ggtakec avatar May 22 '22 06:05 ggtakec

@ggtakec Thank you for your reply.

Although my log does show CURLE_SSL_CACERT error but that isn't really related to my issue, my problem it is that s3fs adds the bucket name to the url which already includes it:

curl_util.cpp:prepare_url(250): URL is https://cse-cic-ids2018.s3.amazonaws.com/cse-cic-ids2018/
curl_util.cpp:prepare_url(283): URL changed is https://cse-cic-ids2018.cse-cic-ids2018.s3.amazonaws.com/
...
curl_util.cpp:url_to_host(327): url is https://cse-cic-ids2018.s3.amazonaws.com

I know this is how s3fs works, my command is wrong and I should remove the bucket name from the url argument.

I'm not reporting a bug, but trying to request a feature to allow passing the full bucket url and perhaps ignore the bucket name argument.

hMED22 avatar May 22 '22 14:05 hMED22

The title and description weren't clear enough and I made it worst by adding the logs of the commands that I expected to work, I was just trying to follow the issue template.

hMED22 avatar May 22 '22 14:05 hMED22

By leaving url option(url=https://cse-cic-ids2018.s3.amazonaws.com/) unspecified, I think https://cse-cic-ids2018.s3.amazonaws.com/ will be used internally. Please try it.

ggtakec avatar May 24 '22 13:05 ggtakec