spec plugin reimplementation

[markus2330]

• [ ] cleanup of copied metadata #670 (see also #4407)
• [ ] make and document a consistent query language: what happens with overlapping globs(_ and #))?
• [ ] specify and document semantics for defaults over globs #3954
• [ ] Make sure availability by default #690

@kodebach did I miss something to be done for spec plugin?

[kodebach]

• what happens with overlapping globs(_ and #))?

What does this mean? Do you mean the following?

IIRC _ means everything that isn't matched by #, so

[foo/_]
default = 1

[foo/#]
default = 2

would be valid in theory. But in practice, we probably want to make that an error. Because array items should never be mixed with other keys.

• Make sure availability by default

That should be handled by the changed in new-backend.

[kodebach]

One thing that I also again and again think about, which may fit here (if this issue ends in a partial re-design of the specification system):

The current very simple plugin execution of "during get (poststorage) we do A -> B -> C and during set (prestorage) we do C -> B -> A" doesn't really work. The idea is obvious, during kdbGet conversion happen in one direction and during kdbSet we go back to restore the original values.

But during a kdbSet things are more complicated. Think about keys that have been added between kdbGet and kdbSet. To make the backwards conversion work, one of two things must be true:

1. The added keys use the format returned by kdbGet. This really isn't ideal, because for example colors may be represented as decimal uint32 values. A format that no human will be able to understand. A simple kdb set user:/color #ff00ff wouldn't work, because during set the conversion only happens in the other direction.

OR

2. There is some logic in the conversion plugin that recognises unconverted keys during kdbSet and does some appropriate extra work. This also a bad solution, because it complicates plugins and recognising unconverted keys isn't easy (see also #3626).

The best solution IMO would be to use a more fine grained notion of what a plugin does. For example:

• "Checker plugin": Does some checks and reports errors/warnings when they fail. Works only with internal, canonical Elektra representations of values (e.g. 0/1 for booleans, decimal uint32 for colors, ...). Importantly these plugins only have one functional mode: check.
• "Conversion plugin": Converts from various representations to the internal, canonical Elektra representation of a value and back. I'll call the two functional modes of these plugins: normalize and restore.

With this distinction, a backend plugin could

• during kdbGet: (load from storage) call conversion plugins in normalize mode and then checker plugins in check mode
• during kdbSet: call conversion plugins in normalize mode (this handles the unconverted values), then checker plugins in check mode and then conversion plugins in restore mode, (write to storage)

Note Notice how conversion plugins are very similar to what we want to do with the spec plugin (called twice during set)

All of this still has a problem. All "Conversion pluigns" must convert directly to the internal, canonical Elektra representation. This hurts modularity. If this wasn't the case, for example the color plugin could just convert to hexadecimal and let the hexnumber plugin convert to the internal decimal (*). However, this doesn't work, with the system above, because we'd have to know which plugin produces which format and which other plugin can read that.

And that is where this links to the specification system. The color plugin could declare in it's contract that it converts from color to hex (where both color and hex are defined by convention similar to METADATA.ini). The hexnumber plugin would declare a conversion from hex to internal (where internal is a special value for the internal, canonical Elektra representation). With this info a specification with meta:/color would mean we need to load the color and the hex plugin and the backend plugin would know in which order to call them.

Final note

I know this would be a huge amount of work, but depending on where we want to go with this re-implementation it may be the correct way. It would certainly be better to invest more time in a good solution that to have something half-working and more open issues for the future.

---

(*) Not the best example, I know. The hex->dec conversion isn't hard. But you can imagine more complex cases, where the second conversion is not trivial.