lakeFS
lakeFS copied to clipboard
Allow for the lakectl local --parallelism flag to have persistent default values through env vars/config
This ticket requests that a mechanism is provided to set a default value of the --parallelism flag for the lakectl local commands. Currently, it defaults to 25 and must be explicitly specified on the command line in order to change the default.
It would be fantastic to allow an environment variable and/or config file to set the default value here, similarly to how some of the configs here: https://docs.lakefs.io/reference/configuration.html influence the local CLI, and allow for use of env vars to set them.
Motivation: I have seen a big performance improvement in lakectl local when working with lots of smaller files with an increased --parallelism flag. Rather than needing to remember to set this flag each time, wrap the tool in helper scripts, etc. it would be great if there was a way to provide a higher custom default directly to the tool.
First asked about here: https://lakefs.slack.com/archives/C016726JLJW/p1725922712803829