Test files not run in CI from pytorch/pytorch

[clee2000]

The following files do not have a run_tests mentioned in the file but are collected by test/run_tests.py, indicating that they probably aren't being run in CI.

distributed/_shard/test_sharder
distributed/elastic/timer/api_test
distributed/optim/test_apply_optimizer_in_backward
distributed/optim/test_named_optimizer
distributed/test_collective_utils
distributed/test_compute_comm_reordering
dynamo/test_backward_higher_order_ops
dynamo/test_compile
dynamo/test_input_attr_tracking
dynamo/test_sources
functorch/test_logging
lazy/test_bindings
lazy/test_extract_compiled_graph
lazy/test_meta_kernel
nn/test_init
optim/test_lrscheduler                                       should be run through test/test_optim.py instead, as noted in the file
optim/test_optim                                             should be run through test/test_optim.py instead, as noted in the file
optim/test_swa_utils                                         should be run through test/test_optim.py instead, as noted in the file
test_cuda_expandable_segments
test_hub
test_model_exports_to_core_aten                  

Being on this list does not mean the test file is guaranteed to run as I did not check to see if they actually call run_tests, only that it is mentioned somewhere in the file.

[huydhn]

AI:

• Add a linter rule to prevent tests added not handled by run_test.py
• Create a new issue to track which pytest feature are available and write a linter to raise errors when people try to use pytest feature that we don't support. Also need to take into account if pytest is supported internally, and which version

@clee2000 We turn off assert rewriting in pytest. We are in a mix situation where we use pytest to run the test but the class still depends on unittest TestCase, otherwise, advance features like retry or disable tests wouldn't work.

[huydhn]

Just another data point, @DanilBaibak mentioned during the oncall handoff today about yet another dynamo tests not run on OSS test/dynamo/test_recompile_ux.py. The good news is that the issue has been fixed in https://github.com/pytorch/pytorch/pull/114235

[clee2000]

Adding https://github.com/pytorch/pytorch/pull/115070 for general pytest tracking: "Pytest replaces sys.stdout/stderr by TextIOWrapper instances which do not support fileno()" which might affect some test cases

[clee2000]

linter has also been merged https://github.com/pytorch/pytorch/pull/114882

[huydhn]

Also the distributed dtensor tests running on CPU that @malfet discovers (See https://github.com/pytorch/pytorch/issues/118043 )

[XilunWu]

re-enable DTensor tests on CPU in #118134

[malfet]

List is no longer accurate, but problem still exists, though nobody is actively looking at it at the moment