How to merge two maps recursively as in Helm?

[anbrsap]

Use Case

In the binding there are two (user-provided) maps with arbitrary content, especially nested maps.

Example:

map1:
  a:
    b: 1
    c: 1
  list: [1, 1, 1]
  x: 1
map2:
  a:
    c: 2
    d: 2
  list: [2]
  y: 2

Using spiff++ I want to merge the maps recursively as the following Helm template would do (using sprig's merge function):

merged: {{ merge $.Values.map1 $.Values.map2 | toYaml | nindent 2 }}

Expected result for above example:

merged:
  a:
    b: 1
    c: 1
    d: 2
  list: [1, 1, 1]
  x: 1
  y: 2

Problem

Using the documentation I cannot find a way to achieve that. The merge functionality in spiff++ always ignores additional fields from the second map. The documentation recommends to mention all possible fields in the template (leftmost argument to merge), but I don't want to build any knowledge about the fields into the template.

[anbrsap]

Now I found it (after hours of trial-and-error):

merged: (( ___.map2 ___.map1 ))

If in the documentation the section heading (( map1 map2 )) would have included e.g. Concatenation of Maps, I would have recognized it immediately. But I was looking for merge and therefore found the merge function and keyword docs, which don't work for my case.

[anbrsap]

No, map concatenation doesn't work recursively. 😠

The result for my initial example would be:

merged:
  a:
    b: 1
    c: 1
    # d: 2 is missing
  list: [1, 1, 1,]
  x: 1
  y: 2

[mandelsoft]

Sorry for my missing answer, I was busy the last few weeks with another project and forgot to look into the spiff project.

The problem with the docu is, that this kind of concatenation works with maps but also for lists and strings. In the table of contents there is an entry (( map1 map2 )). I thought this should be sufficient to look at it when browsing the topic list. Those topics never have more text in the title. Do you think it would be enough to explicitly add the term merging of maps into the description text? Then, a text search in the browser would find it.

[mandelsoft]

BTW: if need more control over the merge of fields with type mismatches, you can also use an own merge lamda expression, that handles then cases as desired:

for example:

util:
  <<<: ((&temporary))
  merge: (( |a,b|->sum[b|a|s,k,v|->s ((type(a.[k]) == "map" ? {k=_(a.[k], v)} :{k=v}) || {k=v})] ))

map1:
  a: a
  b: b
  c:
    a: ca
    b: cb
  d: d

map2:
  a: 2a
  c:
    a: 2ca
    c: 2cc
  d:
    a: 2da
  e: 2e


merge: (( map1 map2 ))
lambda: (( util.merge(map1,map2) ))

It provides the following result:

map1:
  a: a
  b: b
  c:
    a: ca
    b: cb
  d: d
map2:
  a: 2a
  c:
    a: 2ca
    c: 2cc
  d:
    a: 2da
  e: 2e
merge:
  a: 2a
  b: b
  c:
    a: 2ca
    c: 2cc
  d:
    a: 2da
  e: 2e
lambda:
  a: 2a
  b: b
  c:
    a: 2ca
    b: cb
    c: 2cc
  d:
    a: 2da
  e: 2e

As you can see, the (( map1 map2 )) does not recursively merge, in c the value for b from map1 is missing. it just overwrites the the value of c.

The merge lambda only overwrites if the value types do not match, merging a map with a non map just overwrites the map.

This seems to be an important function, and I'll try to add an appropriate built in.

[mandelsoft]

If you want to can try the version from the dev branch. It provides an updated docu and a new function deepmerge.

[anbrsap]

Those topics never have more text in the title.

That way it's not easy for someone not already familiar with Spiff's syntax to find the right section. I would prefer descriptive captions like in the Go language spec.

Do you think it would be enough to explicitly add the term merging of maps into the description text? Then, a text search in the browser would find it.

Not sure. The current description already contains "hereby entries will be merged". Still, when I searched for "merge" I didn't recognize it due to the many other hits.

[mandelsoft]

I just took the structure from the original project. But I understand the problem, I'll add a dedicated section for merging with the next release