test-infra
test-infra copied to clipboard
Codemod service to disable and mark slow tests directly in PyTorch code base
The mechanism to disable a test is sometimes "abused" to keep a test disable for a long time without any owner to work on fixing it. Although, this helps mitigate an flaky test well, it has several drawback:
- It makes thing harder to keep track, a test that is disabled directly in the code is much easier to see
- Closing a disabled issue is tricky https://github.com/pytorch/test-infra/issues/4980
Ideally, we want to keep the current mechanism but limit it only to recently flaky tests, i.e. within 2 weeks. Anything older than that and has not yet been addressed should be disabled directly in the code. Doing this manually doesn't scale, so we need a better tool for this like a codemod service of sort.
cc @clee2000 @malfet @kit1980
AI: It's better to be able to keep the test as XFAIL if possible or a special decorator like skipifFlaky