Configuration merge issue

[ryszard-suchocki]

Hello, I would like to report an issue (maybe it is a feature) with the configuration merge process. My setup consists of a few directories and files. I.E. I have a dedicated folder for repositories config (repos.d) and profile templates (profiles.d). Repo config can inherit another repo config to allow easily choose whether I want to copy backed up data to repo1 or repo2. This works as expected. But when the template profile has a i.e backup section that defines run-before and run-after actions (list of actions) and I add run-before or run-after entries in a specialized profile (derived from template), my changes overload template action. I expect that I can extend configuration, but currently, there is only overload/shadow - whatever you call it. There is some workaround - in the derived profile define multiple empty list items and add desired action/hooks at the end.

template	derived
backup: run-before: - action1 - action2	backup: run-before: - action3 - action4

results invoking the action3 and action4 only (no action1 or action2)

template	derived
backup: run-before: - action1 - action2	backup: run-before: - - - action3 - action4

results invoking action1, action2, action3, action4.

[jkellerer]

@ryszard-suchocki , thanks for reporting this problem.

The intended implementation is:

• Inheriting from another profile: Base profile defines defaults that can be overridden. No merging takes place on individual parameters (e.g. lists are replaced).
• Defining the same profile in multiple includes: Profiles are partially defined in multiple files and are fully merged together. ~~List items should append to each other (within the same profile only)~~ ; Only objects trees merge, lists are treated like values and are replaced, since this happens inside the config parsing library there's no easy path to merge lists.

Can reproduce the problem.

[ryszard-suchocki]

Maybe some kind of switch, that will allow choosing whether to overload or merge? I mean such change is somewhat breaking change.

[creativeprojects]

Maybe some kind of switch, that will allow choosing whether to overload or merge? I mean such change is somewhat breaking change.

That's right, maybe some users are relying on this bug, or tweaked their configuration to make it work with it.

Now, would it be better to have an opt-in or an opt-out switch?

[jkellerer]

I’d keep the bug in version 1 config (the default when no version is specified) if it is caused by inheritance and fix it for version 2 and above.

Includes always merge since this is required to allow partial config definition in multiple files but this happens prior to inheritance processing. If the bug is caused by this (in viper lib as this is doing it) then it must be fixed.

We could define a syntax to add to parameters (e.g. property+=[…]) then a base can be defined and derived profiles can extend lists if needed.

[jkellerer]

Could identify where the bug is, it is in mapstructure. It creates lists (arrays/slices) only when not already existing but inheritance is implemented by parsing the config multiple times. Once lists exist they are treated like objects (e.g. list.idx0 inherits list.idx0 from base).

Can be fixed with a DecodeHookFunc

[jkellerer]

The additional callable config templates #115 in ConfigV2 may provide a way to append to list items while #108 will ensure that lists are no longer merged in an unintentional way.

Configs in version 1 format (without a version flag) will not be changed in any way (the 2 PRs are for version 2 and later)

[ryszard-suchocki]

Let me test before close ;)

[creativeprojects]

That was automatically closed when merging the PR

[ryszard-suchocki]

I tested some the new functionality (resticprofile version 0.18.0-dev commit ""). The new markup "..." works OK.Config v2 changes a lot about how to use resticprofile. Unfortunately, there is no option to append or replace items to the inherited "list" (mixins append works OK!). Please have a look on my config.

Template:

version: 2
mixins:
  kvm-backup:
    default-vars:
      VM: "CHANGE_ME"
      LEVEL: "incr"
      SCHEDULE: "Mon..Fri 8,12,16:00"
    backup:
      run-before...: 
      - '{{ .ConfigDir }}/misc.d/unifi-backup.sh -a backup -m $LEVEL $VM'
      schedule...: 
      - $SCHEDULE
      source...:
      - /storage/kvm/instances/$VM
      tag...:
      - kvm
      - $LEVEL
      - $VM
    retention:
      tag:
      - kvm
      - $LEVEL
      - $VM
    env...:
    - backup_level: $LEVEL
    - domain: $VM

Derived config

{{ $DOMAIN := "MY_VM_NAME"}}
version: 2
profiles:
  {{ $DOMAIN }}:
    use:
    - name: global-defaults
    # notifiers
    - name: notify-telegram
    - name: notify-goalert
    # pre-backup
    - name: kvm-backup
      VM: "{{ $DOMAIN }}"
      SCHEDULE: "Mon..Fri 8,12,16:15"
      LEVEL: "incr"
    # retention
    - name: std-retention
    # repository
    - name: repo-nas01
      DISTINCT: "{{ $DOMAIN }}"
    # replica
    copy:
      use:
      - name: repo-nas02
        DISTINCT: "{{ $DOMAIN }}"

  
  {{ $DOMAIN }}-Diff:
    inherit: {{ $DOMAIN }}
    use:
    - name: kvm-backup
      VM: "{{ $DOMAIN }}"
      SCHEDULE: "Fri 20:00"
      LEVEL: "diff"
    backup:
      tag:
      - newtag # NO REPLACE, NO APPEND - M.I.A.
      - newertag # M.I.A.

Result (produced by "show" command)

global:
    default-command:          snapshots
    restic-binary:           
    ... HERE IS MY GLOBAL CONFIG

profile MY_VM_NAME-Diff:
...
NOTHING INTERESTING
...

    env:
        backup_level:  diff
        domain:       MY_VM_NAME
        tmp:           /tmp

    backup:
...
NOTHING INTERESTING
...
        tag:         kvm
                     diff
                     MY_VM_NAME

    retention:
...
NOTHING INTERESTING
...
        prune:         true
        tag:           kvm
                       diff
                       MY_VM_NAME

    copy:
...
NOTHING INTERESTING
...

schedule MY_VM_NAME-Diff-backup:
    profiles:  MY_VM_NAME-Diff
    run:       backup
    schedule:  Fri 20:00

[jkellerer]

Thanks a lot for the feedback.

The fact that inheritance doesn't work with append is more a technical limitation of the current implementation than a design decision.

If we'd change how inheritance works it could be made possible to support this. Will investigate how much effort it is. If we support it for version 2 then it must be done prior to the release of 0.18 or it needs to wait for a version 3 format.

[creativeprojects]

If we support it for version 2 then it must be done prior to the release of 0.18 or it needs to wait for a version 3 format.

I think configuration version 2 is not an urgent thing. The version 1 can live with 0.18 (or more) for now.

I'd rather make a clean version 2 and then keep it as long as possible: because it means the users will have to upgrade their configuration only once and be done with it.

In fact I was thinking the configuration version 2 would probably make a v1.0.0

(unless you disagree?)

[jkellerer]

Agree.. this means we should mark config version 2 as preview in the documentation and this allows to fix all problems we may find including breaking changes.

[creativeprojects]

That's right, I should start adding some information about version 2 preview in the documentation

[chymian]

That's right, I should start adding some information about version 2 preview in the documentation

meanwhile, is there a raw description of that format available now?

[creativeprojects]

Yes, there's a discussion here: https://github.com/creativeprojects/resticprofile/issues/80