pypsa-eur icon indicating copy to clipboard operation
pypsa-eur copied to clipboard

use `snakemake.params` to auto re-run due to config changes

Open fneum opened this issue 4 years ago • 0 comments

Closes # (if applicable).

Changes proposed in this Pull Request

Use the params: section in rule definition to keep track of changed settings in config.yaml.

https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#non-file-parameters-for-rules

The goal is that rules for which parameters have changed rerun automatically.

This can currently be done with:

https://snakemake.readthedocs.io/en/stable/project_info/faq.html#how-do-i-trigger-re-runs-for-rules-with-updated-code-or-parameters

snakemake -n -R `snakemake --list-params-changes`

or

snakemake -R $(snakemake --list-params-changes)

This will be even more convenient, once this snakemake PR is released:

https://github.com/snakemake/snakemake/pull/1107

snakemake --rerun-params-changed

or

snakemake --rerun-changed

(which would also include code changes)

It seems to work very nicely to pass lists/dicts as params, which would help us keep config "blocks".

This PR is not complete yet, but a proof-of-concept for a selection of config settings. We can discuss from here and complete later.

Checklist

  • [ ] I tested my contribution locally and it seems to work fine.
  • [ ] Code and workflow changes are sufficiently documented.
  • [ ] Newly introduced dependencies are added to envs/environment.yaml and envs/environment.docs.yaml.
  • [ ] Changes in configuration options are added in all of config.default.yaml, config.tutorial.yaml, and test/config.test1.yaml.
  • [ ] Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/configuration.rst and doc/tutorial.rst.
  • [ ] A note for the release notes doc/release_notes.rst is amended in the format of previous release notes.

fneum avatar Dec 04 '21 18:12 fneum