stackdriver-prometheus-sidecar icon indicating copy to clipboard operation
stackdriver-prometheus-sidecar copied to clipboard

Flaky test: TestTailFuzz

Open jkohen opened this issue 7 years ago • 2 comments

This test failed during presubmit, and it's unrelated to my PR (which was documentation only): https://travis-ci.com/Stackdriver/stackdriver-prometheus-sidecar/builds/94064905

@fabxc how can we investigate this?

jkohen avatar Dec 07 '18 23:12 jkohen

The tailer keeps retrying to read more data until its context get canceled. Currently the test waits a fixed amount of time until it cancels after writing the last record. My guess is that the tailer just hasn't read everything yet before the writing goroutine terminates it.

I tried to reproduce this by running the test at high parallelism to cause contention but could not reproduce it. Possibly the process gets completely stalled for longer on Travis. I'd probably just raise the timeout for now. While it's not a 'proper' fix, anything that coordinates explicitly between reader and writer would seem to kill the fuzz part of the test a bit.

fabxc avatar Dec 11 '18 12:12 fabxc

Reopening. I fixed one issue, but while working on that fix, I noticed another flake: panic: write /tmp/test_tail370907351/00000059: file already closed. I confirmed that issue existed before PR #138 .

jkohen avatar Aug 02 '19 20:08 jkohen