Sanitize creds passed in netloc (per Section 3.1 of RFC 1738)?

[cisaacstern]

In https://github.com/pangeo-forge/pangeo-forge-recipes/pull/167, we use urllib.parse.urlparse to prevent query string secrets from appearing in logger output and cached file names.

@jbusecke is working on a recipe for data which reside on a server implementing <user>:<password> authentication according to Section 3.1 of RFC 1738. urlparse recognizes these creds as part of the netloc

# In:
from urllib.parse import urlparse
urlparse('protocol://USERNAME:[email protected]:/path/to/data.nc')
# Out:
ParseResult(scheme='protocol', netloc='USERNAME:[email protected]:', path='/path/to/data.nc', params='', query='', fragment='')

but AFAICT doesn't provide a builtin method for further parsing netloc into a dict of {"creds": x, "host": y}.

@rabernat, is it worth adding a feature to support parsing out this type of authentication? Or perhaps such a feature would be superseded by the upcoming https://github.com/pangeo-forge/pangeo-forge-recipes/pull/245 and is not worth the effort? If the latter, Julius and I can probably hack around his current recipe to prevent credentials appearing in cached file names.

[rabernat]

We should discourage this style of providing credentials (as part of the URL) and instead try to include these in fsspec_open_kwargs. Since it is an fsspec-based interface that should be possible. The IRODSFileSystem class has arguments for user, password etc.

https://github.com/xwcl/irods_fsspec/blob/5adbe4e3e7302b093afd6321c0279b7fc0401e7e/irods_fsspec/init.py#L32-L34

[cisaacstern]

We should discourage this style of providing credentials (as part of the URL) and instead try to include these in fsspec_open_kwargs

👍 Closing because this solution should work for the current recipe. We can reopen if it does not, or if another use case arises for this.

[rabernat]

Have you confirmed that it actually does work?

[cisaacstern]

Have you confirmed that it actually does work?

Oops, no. It just really seemed like it will work and I got over-eager at the possibility of closing an issue. 🤭 Will reopen and close again once (if!) I confirm it works.