[WiP] Assertions based on strategic merge patch (partial list comparisons)

[porridge]

Why we need this

IMHO one of the most pressing issue in kuttl is the problem with assertions on lists (arrays, slices, whatever). To match a list:

• all elements must be listed
• they must be exactly in the same order

For example with status.conditions, depending on the operator which writes the list, this might be impossible to achieve (e.g. if ordering is random or some conditions are present or absent inconsistently.

See #76 for details.

What this PR does

During a recent team hackathon I took a stab at finding out how easy it would be to use the strategic merge patch machinery in order to relax the requirements mentioned above. We basically treat the assertion as a strategic merge patch, apply it to the object at hand, and the assertion passes if and only if the resulting object does not differ from the original.

Credit for the idea goes to @misberner

To be completely explicit, this is just a prototype, but can serve as an inspiration to someone who wanted to take this further.

[dwerder]

That would be really helpful. Whats needed to get this merged?

[porridge]

I think more testing and cleaning up the code would be necessary.

Since this has plenty of potential to trip people up in some corner cases I didn't think about, we should probably consider some more gentle rollout strategy than just wholesale replacing the old comparison mechanism with the new one. Perhaps a toggle in a TestAssert? :thinking:

[chipzoller]

We (Kyverno) would be very interested in having this. The Policy Report format we use outputs an array/object within the CR and that ordering is not guaranteed. We will want to be able to test that a single object that looks like X definition occurs somewhere in that list.

[anderssonw]

We would also love for this to work. We have loads of partial list comparisons in our tests, which for some reason only sometimes fails, probably due to some race condition we are not interested in testing for the tests where we get fails.

[haarchri]

@porridge can we help with something ? we very interested out of crossplane project in this feature ;)

[porridge]

@porridge can we help with something ?

I think these comments still apply @haarchri