matcher icon indicating copy to clipboard operation
matcher copied to clipboard

Several matchers (unordered, containsAll) don't report which element is missing

Open spkersten opened this issue 5 years ago • 2 comments

With matchers like unordered and containsAll, if the value that it is matched against has a different length (or is shorter in case of containsAll) than the expected value, the output of a failed test is:

Expected: equals [
            many items
            ...
          ] unordered
  Actual: ExpandIterable<List<Foo>, Foo>:[
           many items
            ...
          ]
   Which: has too few elements (39 < 40)

Because the lists are truncated it is not even possible to see what is the mismatch. It would be more useful if the output was the elements that are missing/superfluous, like it does when the lengths are the same.

spkersten avatar May 08 '20 10:05 spkersten

I think they are designed to bail out as soon as possible rather than spend (potentially a lot) of time working on finding multiple reasons for a mismatch. I suspect if we dig on this we'll find a tension between the needs of different users and I'd strongly prefer not to introduce a flag or something. I think we might almost be able to justify increasing the time complexity in more cases if we expect the match case to be more common than the mismatch case, but I do worry about cases like isNot(equals(someOtherList)).

I also worry that there may be two goals here which aren't aligned. One goal is to get a quick and hopefully actionable report of problems. Another goal is to deeply debug some problem. In the former case the very furthest we'd go is probably to report the first index of a mismatch even when the lengths don't match. In the latter case you might want more than that, maybe a full diff algorithm to be able to tell the difference between a single item that was inserted, vs completely disparate lists.

I'll keep this open to consider, but I don't anticipate any short term changes. Thanks for the feedback!

natebosch avatar May 08 '20 22:05 natebosch

I think performance concerns don't apply as strongly to each of them. I can image isNot(equals(someOtherList)) being done commonly. However, I wonder how common isNot(containsAll(elements)) or isNot(unordered(elements)) are.

Another approach (probably requiring a lot more refactoring) could be to only construct the message when the expectation fails. Or, to construct it lazily.

spkersten avatar May 09 '20 08:05 spkersten