Suppress string diffs when actual/expected differ too much

[papandreou]

Sometimes the string diffs are just noise, eg. when either actual or expected is the empty string. Certain cheaters exploit the fact that you can specify '' as the expected value and copy paste the diff output into your test and remove the minus signs. If we want to keep that ability we could at least omit the diff when actual is the empty string.

We could also consider a heuristic based on the Levenshtein distance?

[gustavnikolaj]

I had the opposite happen to me when rewriting some tests with unexpected express. It gave some very awkward output, which is comparable to the following example:

unexpected on master $ node -e 'require("./")({key: "Lorem ipsum dolor sit amet, consec
tetur adipiscing elit. Sed vel laoreet nisl. Integer commodo dui sed lacus mattis, eu rh
oncus tortor cursus. Duis sed fringilla leo. Aliquam tempus quis ante et malesuada. Cum
sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Cum so
ciis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Suspendi
sse ullamcorper vulputate ligula non vulputate. Suspendisse pharetra suscipit nibh. Ut p
haretra orci ligula, nec facilisis orci bibendum vel. Sed at metus posuere, ornare nulla
a, rutrum massa."}, "to equal", {key: "Lorem ipsum dolor sit amet, consectetur adipiscin
g elit. Sed vel laoreet nisl. Integer commodo dui sed lacus mattis, eu rhoncus tortor cu
rsus. Duis sed fringilla leo. Aliquam tempus quis ante et malesuada. Cum sociis natoque
penatibus et magnis dis parturient montes, nascetur ridiculus mus. Cum sociis natoque pe
natibus et magnis dis parturient montes, nascetur ridiculus mus. Suspendisse ullamcorper
vulputate ligula non vulputate. Suspendisse pharetra suscipit nibh. Ut pharetra orci lig
ula, nec facilisis orci bibendum vel. Sed at metus posuere, ornare nulla a, rutrum massa
. Foo."})'

/home/gno/Projects/unexpected/lib/Unexpected.js:812
            throw clonedError;
                  ^
Error: 
expected { key: 'Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vellaoreet
nisl. Integer commodo dui sed lacus mattis, eu rhoncus tortor cursus. Duis sed fringilla
leo. Aliquam tempus quis ante et malesuada. Cum sociis natoque penatibus et magnis dis p
arturient montes, nascetur ridiculus mus. Cum sociis natoque penatibus et magnis dis par
turient montes, nascetur ridiculus mus. Suspendisse ullamcorper vulputate ligula non vul
putate. Suspendisse pharetra suscipit nibh. Ut pharetra orci ligula, nec facilisis orci
bibendum vel. Sed at metus posuere, ornare nulla a, rutrum massa.' }
to equal { key: 'Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vel laoree
t nisl. Integer commodo dui sed lacus mattis, eu rhoncus tortor cursus. Duis sed fringil
la leo. Aliquam tempus quis ante et malesuada. Cum sociis natoque penatibus et magnis di
s parturient montes, nascetur ridiculus mus. Cum sociis natoque penatibus et magnis dis
parturient montes, nascetur ridiculus mus. Suspendisse ullamcorper vulputate ligula non
vulputate. Suspendisse pharetra suscipit nibh. Ut pharetra orci ligula, nec facilisis or
ci bibendum vel. Sed at metus posuere, ornare nulla a, rutrum massa. Foo.' }

{
  key: 'Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vel laoreet nisl. I
nteger commodo dui sed lacus mattis, eu rhoncus tortor cursus. Duis sed fringilla leo. A
liquam tempus quis ante et malesuada. Cum sociis natoque penatibus et magnis dis parturi
ent montes, nascetur ridiculus mus. Cum sociis natoque penatibus et magnis dis parturien
t montes, nascetur ridiculus mus. Suspendisse ullamcorper vulputate ligula non vulputate
. Suspendisse pharetra suscipit nibh. Ut pharetra orci ligula, nec facilisis orci bibend
um vel. Sed at metus posuere, ornare nulla a, rutrum massa.' // should equal 'Lorem ipsu
m dolor sit amet, consectetur adipiscing elit. Sed vel laoreet nisl. Integer commodo dui
 sed lacus mattis, eu rhoncus tortor cursus. Duis sed fringilla leo. Aliquam tempus quis
 ante et malesuada. Cum sociis natoque penatibus et magnis dis parturient montes, nascet
ur ridiculus mus. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur
 ridiculus mus. Suspendisse ullamcorper vulputate ligula non vulputate. Suspendisse phar
etra suscipit nibh. Ut pharetra orci ligula, nec facilisis orci bibendum vel. Sed at met
us posuere, ornare nulla a, rutrum massa. Foo.'






                                                             // -Lorem ipsum dolor sit a
met, consectetur adipiscing elit. Sed vel laoreet nisl. Integer commodo dui sed lacus ma
ttis, eu rhoncus tortor cursus. Duis sed fringilla leo. Aliquam tempus quis ante et male
suada. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus
mus. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mu
s. Suspendisse ullamcorper vulputate ligula non vulputate. Suspendisse pharetra suscipit
nibh. Ut pharetra orci ligula, nec facilisis orci bibendum vel. Sed at metus posuere, or
nare nulla a, rutrum massa.






                                                             // +Lorem ipsum dolor sit a
met, consectetur adipiscing elit. Sed vel laoreet nisl. Integer commodo dui sed lacus ma
ttis, eu rhoncus tortor cursus. Duis sed fringilla leo. Aliquam tempus quis ante et male
suada. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus
mus. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mu
s. Suspendisse ullamcorper vulputate ligula non vulputate. Suspendisse pharetra suscipit
nibh. Ut pharetra orci ligula, nec facilisis orci bibendum vel. Sed at metus posuere, or
nare nulla a, rutrum massa. Foo.
}
    at [eval]:1:14
    at Object.<anonymous> ([eval]-wrapper:6:22)
    at Module._compile (module.js:456:26)
    at evalScript (node.js:565:25)
    at startup (node.js:80:7)
    at node.js:935:3

The colors are missing from the above, but the spacing is accurate.

@papandreou and I talked about using Levenshtein distances to meassure this.

var score = levenshtein(actual, expected) / Math.max(actual.length, expected.length);

For two very similar documents, the score would be close small. For very similar long strings, the above output is not very useful - it would be more useful with a unified patch diff, where the significant part of the text is hidden, because it's equal anyway. Something like:

....
Ut pharetra orci ligula, nec facilisis orci bibendum vel.
-Sed at metus posuere, ornare nulla a, rutrum massa.
+Sed at metus posuere, ornare nulla a, rutrum massa. Foo

For two very different strings, the score would be higher, and then the value of a diff like the above would be propertionally smaller.

The calculation of the score should be normalized somehow.

[gustavnikolaj]

When the above example is used in a to satisfy assertion in unexpected-express, you get the complete diff, as well as the output from to equal. The diff is in the same block as the // should equal ... stuff, and thus indented as much.

[alexjeffburke]

We did a very basic thing along these lines for the <string> "starts with"/"end with" assertions so there is definitely precedent for this now.

I'd guess one of the difficultues is around plugins - with, for example, the unexpected-messy based ones which implement a large amount of custom rendering, would there be a negative impact of a general solution to this?

[sunesimonsen]

I think we should look into this as I think it would improve a lot of situations.

I have personally experienced both, where a huge diff with a small difference would have been super useful and the case where you just get a noise diff.

[sunesimonsen]

@alexjeffburke we are only talking about tweaking the string diff.