test-infra icon indicating copy to clipboard operation
test-infra copied to clipboard

Codemod service to disable and mark slow tests directly in PyTorch code base

Open huydhn opened this issue 11 months ago • 1 comments

The mechanism to disable a test is sometimes "abused" to keep a test disable for a long time without any owner to work on fixing it. Although, this helps mitigate an flaky test well, it has several drawback:

  • It makes thing harder to keep track, a test that is disabled directly in the code is much easier to see
  • Closing a disabled issue is tricky https://github.com/pytorch/test-infra/issues/4980

Ideally, we want to keep the current mechanism but limit it only to recently flaky tests, i.e. within 2 weeks. Anything older than that and has not yet been addressed should be disabled directly in the code. Doing this manually doesn't scale, so we need a better tool for this like a codemod service of sort.

cc @clee2000 @malfet @kit1980

huydhn avatar Mar 02 '24 07:03 huydhn

AI: It's better to be able to keep the test as XFAIL if possible or a special decorator like skipifFlaky

huydhn avatar Mar 12 '24 21:03 huydhn