lakectl / Python SDK: Support passing path to config file through environment variable

[AdrianoKF]

Would it make sense to allow specifying the path to the lakectl config file through an env var (e.g., LAKECTL_CONFIG) in addition to passing it through a command-line argument?

The current behavior of either passing the path on the command line or overwriting individual settings through env variables is generally useful. However, in the case of config files in non-standard locations, one needs to always take care to pass the path as part of the command line, making it just a bit more cumbersome.

Similarly, the high-level Python SDK could use the same behavior to facilitate config auto-discovery.

Context: During development on our lakefs-spec package, I frequently find myself switching between multiple lakeFS server instances for testing. Another use case I can imagine is CI environments, where it's usually easy to mount secret credentials as files (however, without being able to control the path - which prevents them from easily being put under ~/.lakectl.yaml) and have their path exposed as an environment variable.

If that sounds like a useful addition, I can put together a draft PR.

Slack reference: https://lakefs.slack.com/archives/C016726JLJW/p1709023046347619

[idanovo]

@nopcoder to decide on ENV var and overriding order (flag vs env var)

[idanovo]

We should also document this somehow as this is not part of lakectl.yaml configuration or a global flag

[nopcoder]

Use LAKECTL_CONFIG_FILE to set the configuration file location and ensure that this property does not conflict with future configuration fields (config_file or config.file)

[idanovo]

@Ben-El Re-opening the issue as it was only implemented for lakectl and not yet for the Python SDK