deprecate, rename and check config entries

[FabianHofmann]

Add scheme for deprecating config entries and warn for config validation

This PR introduces specific warning classes for deprecated and invalid configuration entries. A scheme is proposed to deprecate and optionally rename config entries through the config/deprecations.yaml.

Like this snakemake raises:

• a DeprecationConfigWarning for deprecated config entries
• a InvalidConfigWarning for unsupported/invalid config entries which are not included in config.default.yaml

Testing

Added assertions to verify warning classes in:

• test_config_deprecations()
• test_config_invalid_entries()

Open points

• [x] extend this to capture scenario config.

Checklist

• [x] I tested my contribution locally and it works as intended.
• [x] Code and workflow changes are sufficiently documented.
• [x] Changed dependencies are added to envs/environment.yaml.
• [x] Changes in configuration options are added in config/config.default.yaml.
• [x] Changes in configuration options are documented in doc/configtables/*.csv.
• [x] Sources of newly added data are documented in doc/data_sources.rst.
• [x] A release note doc/release_notes.rst is added.

[github-actions[bot]]

Validator Report

I am the Validator. Download all artifacts here. I'll be back and edit this comment for each new commit.

:exclamation: Run failed!

Download 'logs' artifact to see more details.

• master failed in: no_logs_found
• config-deprecations failed in: no_logs_found

Model Metrics

Benchmarks

Comparing config-deprecations (5bbbecb) with master (590b684). Branch is 6 commits ahead and 0 commits behind. Last updated on 2025-02-04 17:17:01 CET.

[fneum]

This is cool!

[fneum]

I think it’s for deprecation. For validation of cong filers there is already built in functionality in snakemake: https://snakemake.readthedocs.io/en/stable/snakefiles/configuration.html#validation

[lkstrp]

Great, then json schema is already there. But this is even better, we don’t need any extra imports and have everything:

• InvalidConfigWarning: Value validation fails
• DeprecationConfigWarning: Key is not in schema

Json schema allows for basic value checks which should be enough for config validation. And also has a deprecation property, which raises those warnings. This needs to be checked with snakemake, but looks like its build on top and should handle that all.

[FabianHofmann]

Gotcha, so perhaps the naming is a bit misleading, the implementation only checks whether a key provided by the user is deprecated or not supported by the default.yaml. That said I am totally open to generalize this and use a more solid json schema from snakemake which might even check values. I am not sure though, how well these can be incorporated with our scnenario configs, but there is definitely a way. I will take a look. Also the question arises whether it is worth to define the scheme now, before we are doing some potential restructuring changes that might also touch the config ( in the preparation of which I made this PR).

[fneum]

@FabianHofmann, yes I see this as two independent steps. Key deprecation can go ahead.

[FabianHofmann]

okay, given the considerations of @lkstrp I would even consider that we leave this PR and make the whole thing via the jsonschema. there my questions would only be

• can the jsonschema indicate deprecations and custom messages for those?
• can we set the condition that all configurations used (also via the scenario management) must be in the jsonschema and a warning is returned if not

@lkstrp since know more about this, any clue if this is supported? from your answer above it seems so

[lkstrp]

@FabianHofmann Yes, I would rather not merge this one.

I checked it quickly, but I think on both the answer is yes. We need to check if snakemake handles all json-schema features. For scenario management I am not sure though