Introduce compositional Lineages

[sdboyer]

As of right now (Jan 3 2022) there is no code, docs, or anything that deals with compositional lineages. The only hint that they're even a thing is the UnarySchema and UnaryLineage struct types.

These absolutely are going to happen. The chief use case that motivated Thema in the first place - Grafana dashboards - absolutely requires them, and i worked out a halfway generic pattern for them back in August (grafana/grafana#38727).

It's "halfway" in the sense that it introduces a basic pattern for getting composing one lineage into another - a compose key on the lineage, which takes a string-templated key of other lineages, which makes injecting the other lineages pretty easy - but that composition is non-invertible (we have no trivial means of mapping the composed fields back out of the parent schema). Which is a problem for generic translations, and one of three issues to be addressed en route to generic compositional lineages.

Issue 1: Translation de/recomposition

Given an instance of a compositional lineage/schema that we want to translate, we must:

• Decompose the instance into the parts owned by itself/, vs. by the lineages it's composed
• Map the sub-instance back to the original form of the composed lineage (so, invert this mapping)
• Perform translation on the original form through the lenses defined on the composed schema
• Map the translated sub-instance back into the parent, folding any lacunae emitted by composed lineage translation into the accumulator as it proceeds

This shouldn't be horrifyingly difficult - it basically means one more layer of indirection and the introduction of a mapping object somewhat like #Lens, so that instead of directly doing and referencing the remapping right there under the compose key, as above, there's a new kind of object that defines how to map in both directions.

The compose key itself needs to stay top-level on the Lineage, as composition objects are essentially arguments to the whole lineage. But these composition mapping objects will likely need to be sequence-level, as composition mechanism can't change schema-to-schema. That's how we maintain backwards compatibility.

Issue 2: Translation destination

Another key issue to be tackled is control over translation distance with composition. That is, while translating to #Latest or #LatestWithinSequence is easy to map from the parent lineage to composed lineages, a key goal of Thema is that each lineage has their own versioning and history, so mapping #Exact from parent to composed lineage makes no sense.

For example, if i say, "Hey #Translate, take this instance to version [1, 4]", that's fine for the parent, but that version may not exist in the composed lineages. Even if it does, it is completely uncorrelated with the parent version, so whatever motivation i as the caller have for picking [1, 4] in the parent lineage cannot possibly hold for composed lineages. (What if [1, 4] is an upgrade in the parent and a downgrade in the composed?)

I think the solution here is reasonably straightforward, though i haven't thought it all out:

• #Translate will need to be able to handle both unary and compositional lineages. In cases where the #SearchCriteria represent a user intent that can meaningfully cross the composition boundary (#Latest and #LatestWithinSequence, as explained above), those criteria are passed down directly to the composed lineage. For other #SearchCriteria like #Exact, we simply ignore composed lineages.
• A new #TranslateComposed type will be introduced, which will allow the caller to explicitly specify behavior for composed lineages.

These two different translation calls will also weave in the first issue, as #Translate will necessarily flatten all emitted lacunae, whereas #TranslateComposed will emit lacunae within a structure that is isomorphic to the parent lineage's input composition structure.

Issue 3: Other forms of composition

The above is what's necessary for the one approach to lineage composition i've been focused on so far - the one that Grafana needs. A chat i had recently with @franklinhu reinforced a longstanding suspicion that there are other composition patterns that may important to support, which may end up changing some of the requirements we're driving at with both of the above.

I'm not overly concerned here, as i think any future composition forms should be easy to accrete onto Thema in a backwards-compatible way. And because i suspect that most things that initially appear to be compositions are either best or at least sufficiently accomplished as layers on top of the base #Lineage. That's how things turned out with CRDs.