Automated editing of configs is messy

[gethvi]

Speaking as someone who personally manages IntelMQ instance I really don't like it when the IntelMQ changes my configs because it messes them up.

The issues are following:

1. When IntelMQ saves the configuration, it doesn't preserve the order of the config (for instance the global block can end up pretty much anywere in the configuration while I like to keep it on the top).

2. When IntelMQ saves the configuration it uses yaml.default_flow_style = None (default). This means that some blocks end up looking like this:

     parameters:
       file: /opt/intelmq/var/lib/bots/file-output/events.txt
       hierarchical_output: false
       single_key: null
   

   and some end up looking like this:

     parameters: {file: /opt/intelmq/var/lib/bots/file-output/events.txt, hierarchical_output: false,
       single_key: null}
   

   This behavior feels random and inconsistent. IntelMQ should use yaml.default_flow_style = False because it prevents the json-like blocks and makes the behavior consistent.

3. It doesn't preserve comments (which is not such a big deal, but still annoying). They are silently discarded.

The first two issues make it really hard to diff what changes were done by the IntelMQ (for example by the upgrade functions). And using git for versioning the config becomes very painful. Generally I believe it is reasonable to expect and require that when IntelMQ changes config it does so with the minimum amount of changes (and certainly without doing a full remix of the configuration).

Proposed solution:

Use strictyaml instead of ruamel library for working with YAML files. strictyaml solves easily all of the issues above.

From my reasearch only the second issue is easily solvable with ruamel.

[sebix]

Thank you to bring these issues to our attention! We must keep the usability for editing configuration files in mind. With the change from JSON to YAML, we made a huge step forward in this direction, but due to some issues with keeping comments, we intentionally postponed activating this feature to later releases.

Regarding the items you mentioned:

1. keeping the items in order: I agree entirely that keeping the order is indeed favourable and should be envisaged.
2. I agree as well, and setting default_flow_style is a good (short-term) solution
3. We've put some effort and research into this issue but achieved no solution in time

Switching to a different YAML-parser is a valid option; if it solves the issues, we should do it. Thanks for your research searching for solutions and finding this option!

However, I would like to postpone all possible solutions to > 2.1, i.e. in 2.2.0, even the comparatively small default_flow_style change, as it still changes central behaviour and needs to be tested well before releasing it.