default s3.Folder 'secure' setting

[ixcat]

trafficstars

Bug Report

(note: iffy if this is a bug, a request for a shift in behavior, or a bug in requirments spec.. filing as a bug for now)

Description

background:

s3.Folder defaults to secure=False (use HTTP instead of HTTPS). minio.Minio client defaults to secure=True.

when configuring a new s3 external, if that external is HTTPS only*, the connection will fail as follows:

Traceback (most recent call last):
  File "./mctest.py", line 14, in <module>
    exists = mc.bucket_exists(bucket)
  File "/opt/ve/lib/python3.8/site-packages/minio/api.py", line 673, in bucket_exists
    self._execute("HEAD", bucket_name)
  File "/opt/ve/lib/python3.8/site-packages/minio/api.py", line 413, in _execute
    return self._url_open(
  File "/opt/ve/lib/python3.8/site-packages/minio/api.py", line 396, in _url_open
    raise response_error
minio.error.S3Error: S3 operation failed; code: AccessDenied, message: Access denied, resource: /redacted-bucket, request_id: 168127CE7C72244C, host_id: None, bucket_name: redacted-bucket

*: tested against djhub provided s3; Previously saw similar issue on an AWS S3 which I thought was an ACL issue and resolved differently, have not cycled back to re-verify

Setting the store config to have 'secure: true' resolves the issue.

Due to the message (AccessDenied), and the fact that the same endpoint/id/key is OK in minio client, it can be a bit confusing to diagnose this as being the 'secure' setting - 1st impression makes it seem like an actual access/security configuration issue.

Since most production cloud services would be TLS enabled, I'm not sure if defaulting to false makes sense - seems more like a special case for local use.

So - proposing:

• confirm desired behavior
• depending on desired, either default to True, or improve docs for TLS enabled reference config (https://docs.datajoint.io/python/admin/5-blob-config.html#configuration - number 2 under 'principles of operation') to contain the 'secure' setting.
• possibly catch/rethrow the AccessDenied error with more details or add additional debug message depending the real issue (e.g. report actual access denied, or TLS), not sure how easy this might be since the error is coming from within minio.

Reproducibility

tested on ubuntu containers with python 3.8; does not appear to be environment specific.

Expected Behavior

This issue is somewhat about what the expected behavior should be.

[guzman-raphael]

@ixcat Thanks for the report and the debug effort. This seems to be the unfortunate product of initial development w/o secure flag and later introducing the secure feature w/o properly reassessing the best default. I agree that this is a bug.

To align with how MySQL connections are done, the best default would probably be some form of 'best available' option. Meaning we should try secure first and fail down to insecure if necessary. We can include better error-handling when patching this for sure.