Potential flaky tests in the project

[wuhongjun15]

Dear developers, Sorry for bordering you, when I built the project I found the following tests can pass in one build but fail in another, so maybe they are flaky tests or they possess some extent of flakiness?

• com.google.common.cache.PopulatedCachesTest#testReplace_populated
• com.google.common.cache.PopulatedCachesTest#testEntrySet_populated
• com.google.common.util.concurrent.ClosingFutureTest$FinishToValueAndCloserTest#testTransform
• com.google.common.util.concurrent.ClosingFutureTest$FinishToValueAndCloserTest#testWhenAllSucceed2_call
• com.google.common.util.concurrent.UninterruptiblesTest#testJoinWithNoWait
• com.google.common.util.concurrent.UninterruptiblesTest#testPutMultiInterrupt

The detailed test reports are in the attached file, I would appreciate it if you can have a look at this problem.

test reports.txt

[cgdecker]

Yes, it does look like these are tests that could be flaky. Some tests like these are more or less necessarily flaky. In some cases there are ways of making flaky tests deterministic, but some of our libraries, particularly those in concurrent and cache among others, are dealing with low-level enough code that it isn't possible to make them deterministic while still actually testing what we're trying to test. So we just try to make them as resilient to flakiness as we can. It's unlikely that we're going to revisit these methods and attempt to make them less flaky.

[jonahgreenthal]

There have been two other issues opened about flaky tests in the past few days (#5864, #5865), both with similar wording to this one. There were few or no other issues about flaky tests in the last several years. Do we have any idea what's up with this trend?

[cgdecker]

@jonahgreenthal Class assignment most likely.

[ShangwenWang]

Sorry for bothering you for so many times. Actually, we are trying to find silent flaky tests in open-source projects. With these identified flaky tests, we wonder if you plan to fix them in the future. If not, what is the major concern? We think these flaky tests may negatively impact the continuous integration process. Thus, fixing them is necessary, more or less. Is your actual concern is the effort consumtion? What about we can have an automatic repairer?

[cgdecker]

It's not something we have time to prioritize because it's not something that causes us a problem; our internal CI is pretty good about understanding test flakiness and dealing with it. It's certainly not something I'd consider "necessary".

If you guys want to send PRs fixing the flakiness for individual methods or classes, we'd be happy to take a look at those. We just generally aren't going to have time to dig into tests that are likely as un-flaky as some person (or more likely several people) managed to get them in a reasonable amount of time, in code that likely cannot be made 100% deterministic, on the chance that we could fix something that isn't a problem for us (and as far as I can tell, hasn't been a problem for anyone in open-source in the 10+ years that Guava has been around). Hope you understand.

[ShangwenWang]

Thanks for your detailed explanation. We will try to propose reasonable fixes in the future.