Automatically increase diff verbosity up to some threshold budget

[alvaroaleman]

Hi there!

First off, thanks for this project, it is super helpful when writing tests!

One thing that regularly bothers me though is that the generated diff gets pruned. One example I have right now is this:

            - 	nil,
            + 	{
            + 		{
            + 			Added:   []*api.ReleaseBuildConfiguration{&{...}},
            + 			Removed: []*api.ReleaseBuildConfiguration{&{...}},
            + 		},
            + 	},
              )

This is somewhat useless. In order to get a more helpful output, I need to change one of the two variables I am comparing to be more similiar to the other, otherwise I won't even see the ouput. What I would like is an option to tell cmp "Do not prune anything from output, just show me the full diff"

[dsnet]

If possible, I'd like to keep the API surface of cmp smaller. I wonder if I can make use of the testing.Verbose flag as a signal to increase the arbitrary limits we chose for how much stuff gets pruned.

[alvaroaleman]

I don't think that is a great idea, because that would mean verbose mode is required to get useful output on failures which effectively means tests need to be run in verbose mode all the time. IMHO how big the diff can be or not is a mostly static attribute of tests (What is the size of the variables you are comparing?) and not a runtime attribute.

[dsnet]

I don't think that is a great idea, because that would mean verbose mode is required to get useful output on failures which effectively means tests need to be run in verbose mode all the time.

Is it that onerous to re-run the test with verbose? Now that you know which test fails, you can re-run that specific failing test case with the "-v" flag.

IMHO how big the diff can be or not is a mostly static attribute of tests (What is the size of the variables you are comparing?) and not a runtime attribute.

I'm not sure what you mean by "mostly static attribute of tests". In my experience, having looked at many diffs produced by cmp, there are several cases where it tries to print the information for some value which has a field that expands to some enormous amount of data (e.g., type information or metadata information). In such cases, the presence of some value that prints to a large amount is often not a "static attribute of tests" but rather an attribute of the types being compared (which may or may not be under the same control as the test author).

I still stand by trying to not having any knobs to control how formatting is done, but am open to trying to make the output better in the general case. Other ideas that might help:

• Be willing to print quite a bit, up until some minimum character budget is exhausted. For example, being willing to print up to 1KiB of text no matter what.
• Give exported fields a larger print budget over unexported fields.

[alvaroaleman]

Is it that onerous to re-run the test with verbose? Now that you know which test fails, you can re-run that specific failing test case with the "-v" flag.

That works locally, but not in CI where ppl often do something like go test ./.... Getting the verbose output of all tests in all packages just to see the full diff of the one test that failed is not great.

In such cases, the presence of some value that prints to a large amount is often not a "static attribute of tests" but rather an attribute of the types being compared (which may or may not be under the same control as the test author).

It is still under the control of the test author, because the test author decides what types to compare or not? The author might not be able to control all types that are part of the comparison, but they know what will end up being part of the comparison.

Be willing to print quite a bit, up until some minimum character budget is exhausted. For example, being willing to print up to 1KiB of text no matter what.

That sounds like a great idea but it feels like that is going to lead to some bikeshedding about what "quite a bit" is? I toyed a bit around with this, something like 15kiB fits comfortably on half of a screen (depends on newlines though I suppose) so this would just move the discussion to how to change that limit?

[dsnet]

It is still under the control of the test author, because the test author decides what types to compare or not? The author might not be able to control all types that are part of the comparison, but they know what will end up being part of the comparison.

Not always. For example, it is reasonable for your test to depend on protobuf messages that you declare. However, the internal implementation of a protobuf message has a pointer to the type information for the protobuf message, which ends up being a massive amount of information when deeply printed (since the type information is the transitive tree of all other type information that the message may depend on).

I toyed a bit around with this, something like 15kiB fits comfortably on half of a screen (depends on newlines though I suppose) so this would just move the discussion to how to change that limit?

Let's start with an arbitrary minimum limit larger than what it is today. Again, I'd like to avoid any knobs to control formatting if possible.

[alvaroaleman]

@dsnet Are you currently working on the arbitrary minimum limit? Otherwise I'd probably like to take a stab at it

[dsnet]

No, I'm not. Feel free to a take a shot at it. Thanks!

[s3rj1k]

https://github.com/google/go-cmp/issues/279#issuecomment-982800952

Is there an example on how to do that?

[karlkfi]

It looks like the defaultReporter uses a formatOptions struct to control verbosity of Diff output. formatOptions even has VerbosityLevel and LimitVerbosity options built-in. But it doesn't look like there's any way to actually configure them, because the defaultReporter just calls formatOptions{}.FormatDiff with no way to populate the options...

https://github.com/google/go-cmp/blob/master/cmp/report.go#L45

Would it be possible to expose access to the formatOptions?

I'd like to be able to use an Option like cmpopts.WithVerbosity(x) or cmpopts.WithFullVerbosity() to pass to individual cmp.Diff calls.

[wagoodman]

I'd like to be able to use an Option like cmpopts.WithVerbosity(x) or cmpopts.WithFullVerbosity() to pass to individual cmp.Diff calls.

This is exactly what I had in mind as well 👍

[julianKatz]

Would love to see this solved. I'm comparing slices of protos and a result that's missing from the "want" or "got" will show up as a single line. That single line is truncated, meaning I can't see most of the contents of the missing proto.