nomenclature icon indicating copy to clipboard operation
nomenclature copied to clipboard

Filter external repos

Open phackstock opened this issue 5 months ago • 0 comments

Closes #326.

This PR adds the feature to filter external repositories using include and exclude filters. Any number of filters combining any attributes can be defined, example:

repositories:
  common-definitions:
    url: https://github.com/IAMconsortium/common-definitions.git/
definitions:
  variable:
    repository:
      name: common-definitions
      include:
        - name: [Primary Energy*, Final Energy*]
        - name: "Population*"
          tier: 1
      exclude:
        - name: "Final Energy|*|*"
  region:
    repository:
      name: common-definitions
      include:
        - hierarchy: [R5, R10]

For the variable section

in the example above we are including:

  1. All variables starting with Primary Energy or Final Energy
  2. All variables starting with Population and with the tier attribute equal to 1

From this list we are then excluding all variables that match "Final Energy||". This means that the final resulting list will contain no Final Energy variables with three or more levels.

For the region section

we are taking only R5 and R10 regions.

Changes

One of the changes that I have made is that all repositories in the definition section need to have the name key-word. This would be a breaking change so all workflow repos that use external repositories would need to be updated. I'd be happy to streamline it again so that just the name is allowed but for code simplicity I opted against that for now.

phackstock avatar Sep 19 '24 12:09 phackstock