Failing test due to no detected IO events in 'System.IO.Tests.FileSystemWatcherTest.ExecuteAndVerifyEvents'

[Sergio0694]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=709665 Build error leg or test failing: System.IO.Tests.Directory_Delete_Tests.FileSystemWatcher_Directory_Delete_SymLink Pull request: https://github.com/dotnet/runtime/pull/103528

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "Deleted event did not occur as expected",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=709665 Error message validated: [Deleted event did not occur as expected] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 6/17/2024 3:57:06 PM UTC

Report

Build	Definition	Test	Pull Request
2505216	dotnet-runtime	System.IO.Tests.File_Changed_Tests.FileSystemWatcher_File_Changed_SynchronizingObject
759698	dotnet/runtime	System.IO.Tests.File_Move_Tests.File_Move_From_Unwatched_To_Watched	dotnet/runtime#105636
759451	dotnet/runtime	System.IO.Tests.Directory_Create_Tests.FileSystemWatcher_Directory_Create_SymLink	dotnet/runtime#105647

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	3

[jozkee]

This is just FileSystemWatcher being flaky in Windows as always... We may try to add more retries and see if that makes it less noisy.

[jakobbotsch]

@Jozkee Can you disable the test until the flakiness is resolved? This is marked blocking-clean-ci so I don't think putting it in a 10.0 milestone is appropriate.

[jozkee]

What makes an issue be tagged as blocking-clean-ci? i.e. how many times does it have to fail for it to not be blocking-clean-ci-optional?

[jozkee]

I'm hesitant on disabling the tests because they are multiple methods, I would have to disable the whole class which seems risky.

[jakobbotsch]

blocking-clean-ci = it is failing in the default pipelines that run on all PRs. blocking-clean-ci-optional = it is failing in non-default pipelines (like libraries-jitstress).

In both cases the failures are disruptive to a lot of people and should be made non-blocking asap, either by resolving the underlying problems, disabling the tests (with expected future investigation), or removing the tests.

I'm hesitant on disabling the tests because they are multiple methods, I would have to disable the whole class which seems risky.

If these tests are flaky isn't that flakiness just going to mask any real failures, making the test not very useful?

[danmoseley]

In the past I've fixed other FSW tests by adding more retries (or firing more events). Retries are easy to add and they should be outer loop if not already.

[jakobbotsch]

We have a ton of open issues about this problem: #103549, #103584, #103630, #103940, #104269, #104564, #104930 It seems to have super high impact. @Jozkee can you please look into resolving this issue with high priority?

[jakobbotsch]

Still seeing failures, e.g. https://dev.azure.com/dnceng-public/public/_build/results?buildId=747310&view=ms.vss-test-web.build-test-results-tab&runId=18893294&resultId=119514&paneView=debug, console log https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-3c8d50d456884ef49b/System.IO.FileSystem.Watcher.Tests/1/console.7ac301db.log?helixlogtype=result.

  Starting:    System.IO.FileSystem.Watcher.Tests (parallel test collections = on [2 threads], stop on fail = off)
    System.IO.Tests.Directory_NotifyFilter_Tests.FileSystemWatcher_Directory_NotifyFilter_LastWriteTime(filter: LastWrite) [FAIL]
      Changed event did not occur as expected
      Stack Trace:
        /_/src/libraries/System.IO.FileSystem.Watcher/tests/Utility/FileSystemWatcherTest.cs(317,0): at System.IO.Tests.FileSystemWatcherTest.ExecuteAndVerifyEvents(FileSystemWatcher watcher, WatcherChangeTypes expectedEvents, Action action, Boolean assertExpected, String[] expectedPaths, Int32 timeout)
        /_/src/libraries/System.IO.FileSystem.Watcher/tests/Utility/FileSystemWatcherTest.cs(198,0): at System.IO.Tests.FileSystemWatcherTest.ExpectEvent(FileSystemWatcher watcher, WatcherChangeTypes expectedEvents, Action action, Action cleanup, String[] expectedPaths, Int32 attempts, Int32 timeout)
        /_/src/libraries/System.IO.FileSystem.Watcher/tests/Utility/FileSystemWatcherTest.cs(168,0): at System.IO.Tests.FileSystemWatcherTest.ExpectEvent(FileSystemWatcher watcher, WatcherChangeTypes expectedEvents, Action action, Action cleanup, String expectedPath, Int32 attempts, Int32 timeout)
        /_/src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.Directory.NotifyFilter.cs(136,0): at System.IO.Tests.Directory_NotifyFilter_Tests.FileSystemWatcher_Directory_NotifyFilter_LastWriteTime(NotifyFilters filter)
           at InvokeStub_Directory_NotifyFilter_Tests.FileSystemWatcher_Directory_NotifyFilter_LastWriteTime(Object, Span`1)
           at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
RetryHelper: retrying FileSystemWatcher_Directory_Changed_LastWrite 1th time of 3: got Changed event did not occur as expected

[jozkee]

Yes, I disabled the ones reported, it was expected that more would come up.

[jeffhandley]

@Jozkee Are there any other tests in this realm that are still active and causing noise, or do you think they're all disabled now? It looks like the hit count has dropped off since you merged in #105254 but I wanted to confirm with you.

If you think we're now clean for the 9.0.0 branch, we can move this to Future and clear the assignee.

[jozkee]

Are there any other tests in this realm that are still active and causing noise

Not that I'm aware of.

we can move this to Future and clear the assignee.

https://github.com/dotnet/runtime/pull/105254 left us with a huge test gap for FileSystemWatcher in Windows. We should find ways to re-enable the tests a reduce their noise. Future is not a good idea, 10.0.0 perhaps.