Allow fromfile config value to be optional

[okcleary]

Is your feature request related to a problem? Please describe.

Pants will throw an error when parsing config given a fromfile config value where the file does not exist. This is a good default behaviour for giving feedback to the user, however this can make some use-cases cumbersome because even goals that do not require the config value are unable to be run when the file is missing.

Given:

[python-repos]
indexes = "@/file/does/not/exist"

Then:

> pants
08:46:09.36 [INFO] Initializing scheduler...
08:46:11.74 [INFO] Scheduler initialized.
08:46:11.76 [ERROR] Error computing value for --indexes in scope 'python-repos' (may also be from PANTS_* environment variables). Caused by:

Failed to read indexes in scope 'python-repos' from file /file/does/not/exist: FileNotFoundError(2, 'No such file or directory')

Describe the solution you'd like

Allow a user to mark a fromfile config as "optional" so when the referenced file does not exist pants treats the config value as simply not being set.

My suggested implementation is to simply use a @? prefix instead of the normal @ to mark it.

Given:

[python-repos]
indexes = "@?/file/does/not/exist"

Then:

> pants
08:48:07.17 [INFO] Initializing scheduler...
08:48:09.59 [INFO] Scheduler initialized.
No goals specified.
Use `pants help` to get help.
Use `pants help goals` to list goals.

Describe alternatives you've considered

Use an env var for fromfile only when needed.

This is a viable alternative as it addresses my problems with using fromfile in the config file. It does however lack the clear visibility that having it in the config file gives, and also does not scope the config to just the project at hand.

Use a cli arg for fromfile only when needed.

This addresses my problems with using fromfile in the config file, but it also means that the cli arg needs to be provided for all pants calls that do need it (annoying devex).

Use an env var in pants.toml instead.

This suffers similar pitfalls, when the env var is missing config parsing fails (this is probably another issue to raise, as the error message gives an attribute error on types.SimpleNamespace instead of something a little more clear like "env var not set").

Don't use pants config at all.

For my specific use-case I can actually get around having to configure anything in pants config at all - it does however require more work to have this running nicely, as opposed to a simple script in the repository to generate a file.

Default/fallback value for a fromfile.

My initial idea for facilitating this was allowing defining a fallback value for when a file does not exist, which would be nicer feature wise but I quickly abandoned when thinking it through due to the apparent complexity of both implementation and API.

Plugin to manage config at runtime or to call hooks pre-config.

Perhaps an ideal solution for my use-case is having pre-config hooks to scripts that can setup state for the execution - I haven't investigated this though and I'm assuming it's a complex problem to solve (because hooks would be configured in config...).

A more targeted solution would be to create a plugin to dynamically extend specific subsystems configurations (e.g python-repos.indexes), but again I haven't investigated this to understand it's complexity or viability.

Additional context

The background to the request, and my immediate use-case, is that I'm using a fromfile config to populate the python-repo.indexes value from a json file. Specifically this json file contains an AWS Codeartefact registry URL that includes the credentials in the URL.

Codeartefact credentials only last for 24 hours and so needs to periodically be refreshed, this is facilitated in our repo by a helper script that is run before pants commands in the makefile. Even though these credentials are short lived, we still do not want them committed to the repository so the generated file is in the gitignore, meaning the default state of the repository has the file missing.

This actually works fine for most contexts, as a user just has to run through some basic setup steps to get around the problem, however it becomes annoying in our pipelines as the file needs to be touched in every context before any pants command is run (including simply running the github actions pants setup action).

Note: I have worked around this problem for my use-case, I've raised this as I believe it to be a useful enhancement to the fromfile config feature.