Switching env.yaml without restarting the Jupyter process.

[edublancas]

Currently, pipeline.yaml looks for an env.yaml file to search for parameters, and the only way to override this is via the PLOOMBER_ENV_FILENAME environment variable.

However, changing the variable isn't practical when JupyterLab runs on a hosted environment because users can't easily shut down the process, modify the variable, and start Jupyter again. This isn't an issue if running the pipeline using the command line but it is if users wan't to switch the parameters and see the new configuration in the injected cell.

I think the simplest way to do this is with an optional field in pipeline.yaml

meta:
    env: env.yaml

Currently, priority is as follows:

1. Value in PLOOMBER_ENV_FILENAME
2. env.{name}.yaml (if pipeline has pipeline.{name}.yaml) format
3. env.yaml

After the change:

1. Value in PLOOMBER_ENV_FILENAME
2. Value in meta.env , if any
3. env.{name}.yaml (if pipeline has pipeline.{name}.yaml) format
4. env.yaml

This is a small change, we just need to modify this function: https://github.com/ploomber/ploomber/blob/a278dd1d4c504f81100d248af57a49329d3388b0/src/ploomber/util/default.py#L236

And allow env in the meta section:

https://github.com/ploomber/ploomber/blob/a278dd1d4c504f81100d248af57a49329d3388b0/src/ploomber/spec/dagspec.py#L625

Note: we need to validate that meta.env has a filename, not a path (e.g., env.something.yaml and not path/to/env.something.yaml) - we already do this fo the env variable: https://github.com/ploomber/ploomber/blob/a278dd1d4c504f81100d248af57a49329d3388b0/src/ploomber/util/default.py#L353

Want do you think of this change, @filipj8? Wanna work on this?

To do:

• [ ] modify default.py
• [ ] add tests for default.py
• [ ] modify dagspec.py
• [ ] add tests for dagspec.py
• [ ] update pipeline.yaml docs
• [ ] update docs section that explain how the env var works

[edublancas]

closing in favor of #690