pangeo-forge-recipes
pangeo-forge-recipes copied to clipboard
Sanitize creds passed in netloc (per Section 3.1 of RFC 1738)?
In https://github.com/pangeo-forge/pangeo-forge-recipes/pull/167, we use urllib.parse.urlparse
to prevent query string secrets from appearing in logger output and cached file names.
@jbusecke is working on a recipe for data which reside on a server implementing <user>:<password>
authentication according to Section 3.1 of RFC 1738. urlparse
recognizes these creds as part of the netloc
# In:
from urllib.parse import urlparse
urlparse('protocol://USERNAME:[email protected]:/path/to/data.nc')
# Out:
ParseResult(scheme='protocol', netloc='USERNAME:[email protected]:', path='/path/to/data.nc', params='', query='', fragment='')
but AFAICT doesn't provide a builtin method for further parsing netloc
into a dict of {"creds": x, "host": y}.
@rabernat, is it worth adding a feature to support parsing out this type of authentication? Or perhaps such a feature would be superseded by the upcoming https://github.com/pangeo-forge/pangeo-forge-recipes/pull/245 and is not worth the effort? If the latter, Julius and I can probably hack around his current recipe to prevent credentials appearing in cached file names.
We should discourage this style of providing credentials (as part of the URL) and instead try to include these in fsspec_open_kwargs
. Since it is an fsspec-based interface that should be possible. The IRODSFileSystem
class has arguments for user
, password
etc.
https://github.com/xwcl/irods_fsspec/blob/5adbe4e3e7302b093afd6321c0279b7fc0401e7e/irods_fsspec/init.py#L32-L34
We should discourage this style of providing credentials (as part of the URL) and instead try to include these in fsspec_open_kwargs
👍 Closing because this solution should work for the current recipe. We can reopen if it does not, or if another use case arises for this.
Have you confirmed that it actually does work?
Have you confirmed that it actually does work?
Oops, no. It just really seemed like it will work and I got over-eager at the possibility of closing an issue. 🤭 Will reopen and close again once (if!) I confirm it works.