CI: Make the e2e test output more readable

[markmandel]

We are outputting lots of text, so it gets truncated on the Cloud Build UI, and you have to revent to using the RAW output, which is much harder to traipse through to find the error you need.

All you get is this little tiny comment at the bottom:

The displayed log content is truncated to 5 MiB. Click 'View raw' to view the full content as plaintext.

Which is also super hard to see 😃

The new e2e matrix is awesome, but even for passing tests, outputs all the output as verbose. It would be ideal if it only did that if the tests fail (so verbose should be off), so we are less at risk out outputting too much information.

[zmerlynn]

If you go into the step output, you can see the whole e2e output. I agree, though, and I struggled to find the right balance - in part because there's a lot of output slicing that goes on, so it's hard to just dump the verbose output of the failed tests.

[markmandel]

If you turn off verbosity in go tests, it will give you the output only if the tests fail. So it should be a quick fix.

[zmerlynn]

If you turn off verbosity in go tests, it will give you the output only if the tests fail. So it should be a quick fix.

I spent about a half day on this a few months ago. The only output that can be trusted is the full, verbose output, allowing all tests to spam, if you want the stdout/stderr output from tests. Otherwise the only reliable output you get is the output from the t.Log interface itself (so e.g. the messages from testify, etc.)

logrus messages related to one test may end up randomly misattributed to another test's output, and there's not a good way to control it outside of running all e2es serially. If you want to see this in action, try running the e2es with the -json flag - the JSON output has a field for which test go test thinks the output is part of, but you can very clearly watch as logrus messages from test A end up attributed to test B. Note that this isn't related to logrus specifically, but anything going to stdout/stderr and not the t.Log() stream (and there's not a good way to force logs globally through t.Log()).

Also importantly, because we run tests in parallel, the current output actually does show you logs relevant to surrounding tests. Even if the log misattribution issue was fixed, if we were to hide output from passing tests, we may miss global interactions.

At the end of the day, I settled on extremely verbose output. Maybe we need a way to pipe the output somewhere else and present it better? Not sure.

ETA: Maybe there's some option where we can write logs somewhere and store them as build artifacts?

[github-actions[bot]]

'This issue is marked as Stale due to inactivity for more than 30 days. To avoid being marked as 'stale' please add 'awaiting-maintainer' label or add a comment. Thank you for your contributions '

[markmandel]

Going to close this for now, it's the best I think we're going to get it with Cloud Build for now.