datadriven
datadriven copied to clipboard
cockroachdb
[WIP] Create a framework to exercise the `go test` output
Prior to this patch, there was no way to exercise the behavior of
-rewrite nor how errors and sub-tests would be reported in the
output of go test -v.
This patch introduces such "reflective" tests, where a test can
compare the output and modified file through running go test
recursively.
This change is 
What do you think? I was missing a way to properly test all the features of the datadriven package. I think this way we can do this.
If you think this is a good idea, I will add unit tests for the features already there - error handling, sub-tests, parsing errors, etc.
Couldn't we change the infrastructure to take a testing.TB and provide an instrumented implementation for the purposes of testing?
That was my first idea actually, but I found it unsatisfying for two reasons.
The first reason is that it's quite a large amount of work. Not just the interface change, which is minor; the implementation of a mock testing.TB is quite the challenge, as there is a complex interplay between goroutines (one per test), parent-child goroutines (subtests) and the channels used to synchronize (t.Fatal/t.Skip use runtime.Goexit to terminate early). Just the mock would implement 30-40% of the testing package, which is quite a lot.
The second reason (and for me the main reason) is that I really want to test the output produced by failing datadriven tests when ran under go test:
- the command line flag parsing
- how the file/lineno reference look like in error messages
- whether the sub-test names are properly assembled (I just found a bug in this area, PR upcoming)
It looks to me like the amount of refactoring / mock injection needed to test these aspects is unreasonably large compared to the (relative) simplicity of the approach I took here instead.
What do you think?
I was thinking the implementation would still use a *testing.T underneath (the one from our test), you'd just intercept some of the calls we care about (Error,Fatal). I might be missing some subtleties though, I'll leave it to you to decide since you spent more time thinking about this. I don't have strong objections given that it's testing code and not even in the main repo. I'll look more closely at the final change.
you'd just intercept some of the calls we care about (Error,Fatal). I might be missing some subtleties though
I'd like to understand more: how do you mock something that properly casts down to *testing.T in some places, but uses a mock in others?
It wouldn't cast down, but it would implement testing.TB. Hm though I guess that doesn't have everything we need (eg Run). We'd need to define our own interface and that will make the API more complicated. Your solution is probably better.
It doesn't have Run and also implementing Fatal and Skip is annoying (goroutines + defers + weird synchronization)