Documentation on object merging seems confusing and contradictory ("fallback", "falls back", "non-object always wins")

[jmm]

trafficstars

Hello,

Even though I believe I understand the behavior, and there's an illustration of it, I can not make sense of the description in "Duplicate keys and object merging":

if you set a key to an object, a non-object, then an object, first the non-object falls back to the object (non-object always wins), and then the object falls back to the non-object (no merging, object is the new value). So the two objects never see each other.

It's not clear to me what it means by:

• "falls back"

• "non-object always wins" (if this was the case, wouldn't the final value be the one non-object?)

• "the object" (especially in concert with the "falls back" terminology)

If my understanding of the actual behavior is correct, then I'd highly recommend rewording that, something like:

if you set a key to an object (# 1), a non-object (# 2), then an object (# 3):

• First # 2 overrides # 1 and the value is non-object.

• Then # 3 overrides # 2 and the value is object (# 3).

• The objects in # 1 and # 3 never see each other and no merging is performed.

The confusion is highlighted by the later section "Config object merging and file merging". Wouldn't the final values in the examples be { a : { y : 2 } } and { a : 42 }?

The "fallback" terminology and example seems to indicate the opposite logic and outcome of the example in "Duplicate keys and object merging" and imply that the later assignments are defaults, not overrides that take precedence.

If...

All merging works the same way

...and...

In HOCON, duplicate keys that appear later override those that appear earlier, unless both values are objects.

...then both of the examples in "Config object merging and file merging" seem wrong.

---

The "non-object always wins" concept seems confusing. That seems to say that in the examples of assigning object, then non-object, then object, that the final value would end up being the non-object.

Wouldn't the behavior be explained more simply and clearly by saying something like:

When a key is assigned a non-object value followed by an object value (or vice versa), there's no merging and the later value takes precedence.

Is there ever a case where you'd do two assignments, one of which is object and the other non-object (in either order), and the later assignment wouldn't prevail regardless of which was the object and which was non-object?

[havocp]

In "Config object merging and file merging" the items are shown backward from how they would be in a file (first priority first rather than last), reordering those bullet lists would probably be helpful and I think would clarify your question about the final values.

The behavior this is trying to explain is that merging values should be associative, that is, if you have values a,b,c, and merge operation (let's call it :star: ), then (a :star: b) :star: c should have the same result as a :star: (b :star: c). Also it is always a binary operation; merging a list of N values is always done by folding (merging them 2 at a time). There's never a case where we look at more than 2 values together and decide to do something different based on that. By the merge operation :star:, I mean merging two objects OR keeping only one of the values (if one value is a non-object). :star: is the method Config.withFallback in the Java code.

Your suggested rewordings seem helpful to me. If you want to take a crack at a PR with some edits, that might help the next person to come along.

[jmm]

Thanks for the (quick) reply! I'm dealing with the great Amazon outage right now, but will read your reply carefully as soon as I get the chance.

[jmm]

In "Config object merging and file merging" the items are shown backward from how they would be in a file

Ahh, ok, thanks. I now understand the usage of "fallback" there. I still can't make sense of its usage in the other section.

reordering those bullet lists would probably be helpful

Hmm, I understand what it's saying now, but it is unfortunate that the order is opposite the examples in the other section. A code snippet would probably help, but then again I found the behavior of withFallback confusing.

The behavior this is trying to explain is that merging values should be associative, that is, if you have values a,b,c, and merge operation (let's call it ⭐ ), then (a ⭐ b) ⭐ c should have the same result as a ⭐ (b ⭐ c).

Ok thanks. I didn't actually understand what you meant until I looked into Config.withFallback and at first I didn't understand how that works either.

I would've thought these would have different outcomes:

(
  object.withFallback(nonObject)
).withFallback(otherObject)

object.withFallback(
  nonObject.withFallback(otherObject)
)

The documentation says it returns:

a new object (or the original one, if the fallback doesn't get used)

But actually when the fallback isn't used it returns a new object that remembers the attempted merge, right?

Your suggested rewordings seem helpful to me. If you want to take a crack at a PR with some edits, that might help the next person to come along.

Ok, thank you. I'll see if I have an opportunity to do that.

[havocp]

So unpacking this tortured text,

if you set a key to an object, a non-object, then an object, first the non-object falls back to the object (non-object always wins), and then the object falls back to the non-object (no merging, object is the new value). So the two objects never see each other. This means that if you have:

foo = { "b" : 42 }
foo = 37
foo = { "a" : 53 }

Then it isn't correct to evaluate this by saying {"a":53}.withFallback(37)={"a":53} and then doing {"a":53}.withFallback(("b":42})={"a":53,"b":42} ... the correct result is just {"a":53}. The unit test for this is https://github.com/lightbend/config/blob/e0984d41b040d64b650c7014108a7db9dd451457/config/src/test/scala/com/typesafe/config/impl/ConfigTest.scala#L223-L244 The implementation in Java is that values can have an "ignores fallbacks" flag that gets set if you fall back to something we can't merge across: https://github.com/lightbend/config/blob/e0984d41b040d64b650c7014108a7db9dd451457/config/src/main/java/com/typesafe/config/impl/AbstractConfigValue.java#L234 and this means we "remember" that the 37 was there, so {"a":53}.withFallback(("b":42})={"a":53}. There are effectively two different kinds of object value, objects that can still merge in fallback objects, and objects that have been flagged as ignoring all fallbacks.

Implementation-wise it's even more complicated because there can also be unresolved substitutions ${foo} and we don't know yet whether they are objects or not. So when doing withFallback for values containing those unresolved values, we may have to remember a whole fallback chain and merge it once all the values are made concrete.

But actually when the fallback isn't used it returns a new object that remembers the attempted merge, right?

Yes I think it may need to return a new object with the ignoresFallback flag set, as implemented (but it may return the original object if ignoresFallback was already set on it).

[jmm]

Thanks for your reply and walking through all of that!

Yeah, initially the withFallback behavior seemed counterintuitive to me, but as I studied it before my previous post I understood why it was implemented that way (some of the reasons at least). Thank you for the additional insight into "ignores fallbacks" and the implications of substitution, that makes sense.