pytest-dev
Add tensorflow and pytorch random seed setting & Update requirements.
Tests and README are included.
Add support for setting tensorflow and pytorch random seed. It can help detect flaky tests due to the randomness, and provide reproducibility.
Reference: [1] Dutta, Saikat, et al. "Detecting flaky tests in probabilistic and machine learning applications." Proceedings of the 29th ACM SIGSOFT international symposium on software testing and analysis. 2020.
Why did you close this? I started reviewing and tidying last night. I think it is a great PR, thank you very much.
Why did you close this? I started reviewing and tidying last night. I think it is a great PR, thank you very much.
I tried adding pytorch and tensorflow requirements to pass the test, but it seems I did something wrong. This PR failed checks.
You didn’t mess anything up. The requirements are compiled on the OS you run requirements/compile.py, in your case macOS. Normally there aren’t any OS-specific packages, indeed across all my 33 open source projects there aren’t. But it seems that tensorflow is split into OS-specific packages, so the requirements currently have tensorflow-macos compiled in. Naturally, this doesn't install on the Linux CI runner.
I will try come up with a solution here, perhaps adding tensorflow to the .txt files without pinning the OS-specific packages.
Maybe you can use compile.py to regenerate these py3*.txt files as before, with I added tensorflow and pytorch in requirements.in.
Thanks for your explanation!
I will work on it when I can. Please do not ping. It is not helpful and you should have no expectation of free service. If you are going to post a message, provide information, such as "I tried using my fork in a project and it worked well" or similar.
Python 3.12 has a segmentation fault, both locally and on CI.
Traceback
Fatal Python error: Segmentation fault
Current thread 0x00007f6e828aeb80 (most recent call first):
Garbage-collecting
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/ast.py", line 52 in parse
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/_code/source.py", line 185 in getstatementrange_ast
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/_code/code.py", line 261 in getsource
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/_code/code.py", line 834 in _getentrysource
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/_code/code.py", line 932 in repr_traceback_entry
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/_code/code.py", line 994 in repr_traceback
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/_code/code.py", line 1064 in repr_excinfo
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/_code/code.py", line 699 in getrepr
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/nodes.py", line 464 in _repr_failure_py
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/nodes.py", line 541 in repr_failure
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/runner.py", line 409 in pytest_make_collect_report
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_callers.py", line 102 in _multicall
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_manager.py", line 119 in _hookexec
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_hooks.py", line 501 in __call__
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/runner.py", line 565 in collect_one_node
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/main.py", line 839 in _collect_one_node
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/main.py", line 976 in genitems
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/main.py", line 981 in genitems
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/main.py", line 813 in perform_collect
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/main.py", line 349 in pytest_collection
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_callers.py", line 102 in _multicall
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_manager.py", line 119 in _hookexec
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_hooks.py", line 501 in __call__
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/main.py", line 338 in _main
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/main.py", line 285 in wrap_session
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/main.py", line 332 in pytest_cmdline_main
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_callers.py", line 102 in _multicall
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_manager.py", line 119 in _hookexec
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_hooks.py", line 501 in __call__
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/config/__init__.py", line 174 in main
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/config/__init__.py", line 197 in console_main
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pytest/__main__.py", line 7 in <module>
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/coverage/execfile.py", line 211 in run
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/coverage/cmdline.py", line 858 in do_run
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/coverage/cmdline.py", line 681 in command_line
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/coverage/cmdline.py", line 970 in main
File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/coverage/__main__.py", line 10 in <module>
File "<frozen runpy>", line 88 in _run_code
File "<frozen runpy>", line 198 in _run_module_as_main
Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special (total: 20)
This occurs reliably and disappears when removing the tensorflow import. But it seems to be part of numpy. I have looked on both projects’ issue trackers and found nothing, and haven’t got the time to chase creating a minimal example. Would you be interested in doing that?
Yes, I'm trying to do so. But I'm not sure how to create a minimal example of this case.
I noticed that except Python 3.12, other versions of CI worked well. With the below dependencies installed:
py311: absl-py==2.1.0,asgiref==3.8.0,astunparse==1.6.3,certifi==2024.2.2,charset-normalizer==3.3.2,coverage==7.4.4,Django==5.0.3,execnet==2.0.2,factory-boy==3.3.0,Faker==24.3.0,filelock==3.13.1,flatbuffers==24.3.25,fsspec==2024.3.1,gast==0.5.4,google-pasta==0.2.0,grpcio==1.62.1,h5py==3.10.0,idna==3.6,iniconfig==2.0.0,Jinja2==3.1.3,keras==3.1.1,libclang==18.1.1,Markdown==3.6,markdown-it-py==3.0.0,MarkupSafe==2.1.5,mdurl==0.1.2,ml-dtypes==0.3.2,model-bakery==1.17.0,mpmath==1.3.0,namex==0.0.7,networkx==3.2.1,numpy==1.26.4,nvidia-cublas-cu12==12.1.3.1,nvidia-cuda-cupti-cu12==12.1.105,nvidia-cuda-nvrtc-cu12==12.1.105,nvidia-cuda-runtime-cu12==12.1.105,nvidia-cudnn-cu12==8.9.2.26,nvidia-cufft-cu12==11.0.2.54,nvidia-curand-cu12==10.3.2.106,nvidia-cusolver-cu12==11.4.5.107,nvidia-cusparse-cu12==12.1.0.106,nvidia-nccl-cu12==2.19.3,nvidia-nvjitlink-cu12==12.4.99,nvidia-nvtx-cu12==12.1.105,opt-einsum==3.3.0,optree==0.11.0,packaging==24.0,pip==24.0,pluggy==1.4.0,protobuf==4.25.3,Pygments==2.17.2,pytest==8.1.1,pytest-randomly @ file:///home/runner/work/pytest-randomly/pytest-randomly/.tox/.tmp/package/1/pytest_randomly-3.15.0-py3-none-any.whl#sha256=33ad92c347928f2b348b0575fe78fb9f5ee21fe8a97540399b386a28a2b4b17f,pytest-xdist==3.5.0,python-dateutil==2.9.0.post0,requests==2.31.0,rich==13.7.1,setuptools==69.1.0,six==1.16.0,sqlparse==0.4.4,sympy==1.12,tensorboard==2.16.2,tensorboard-data-server==0.7.2,tensorflow==2.16.1,tensorflow-io-gcs-filesystem==0.36.0,termcolor==2.4.0,torch==2.2.1,triton==2.2.0,typing_extensions==4.10.0,urllib3==2.2.1,Werkzeug==3.0.1,wheel==0.42.0,wrapt==1.16.0
py311: commands[0]> python -W error::ResourceWarning -W error::DeprecationWarning -W error::PendingDeprecationWarning -W 'ignore:pkg_resources is deprecated as an API:DeprecationWarning' -W 'ignore:datetime.datetime.utcfromtimestamp() is deprecated:DeprecationWarning' -m coverage run -m pytest -p no:randomly tests
============================= test session starts ==============================
While the Python 3.12 CI failed with the below dependencies installed:
py312: absl-py==2.1.0,asgiref==3.8.1,astunparse==1.6.3,certifi==2024.2.2,charset-normalizer==3.3.2,colorama==0.4.6,coverage==7.4.4,Django==5.0.3,execnet==2.0.2,factory-boy==3.3.0,Faker==24.4.0,filelock==3.13.3,flatbuffers==24.3.25,fsspec==2024.3.1,gast==0.5.4,google-pasta==0.2.0,grpcio==1.62.1,h5py==3.10.0,idna==3.6,iniconfig==2.0.0,Jinja2==3.1.3,keras==3.1.1,libclang==18.1.1,Markdown==3.6,markdown-it-py==3.0.0,MarkupSafe==2.1.5,mdurl==0.1.2,ml-dtypes==0.3.2,model-bakery==1.17.0,mpmath==1.3.0,namex==0.0.7,networkx==3.2.1,numpy==1.26.4,nvidia-cublas-cu12==12.1.3.1,nvidia-cuda-cupti-cu12==12.1.105,nvidia-cuda-nvrtc-cu12==12.1.105,nvidia-cuda-runtime-cu12==12.1.105,nvidia-cudnn-cu12==8.9.2.26,nvidia-cufft-cu12==11.0.2.54,nvidia-curand-cu12==10.3.2.106,nvidia-cusolver-cu12==11.4.5.107,nvidia-cusparse-cu12==12.1.0.106,nvidia-nccl-cu12==2.19.3,nvidia-nvjitlink-cu12==12.4.99,nvidia-nvtx-cu12==12.1.105,opt-einsum==3.3.0,optree==0.11.0,packaging==24.0,pip==24.0,pluggy==1.4.0,protobuf==4.25.3,Pygments==2.17.2,pytest==8.1.1,pytest-randomly @ file:///home/runner/work/pytest-randomly/pytest-randomly/.tox/.tmp/package/1/pytest_randomly-3.15.0-py3-none-any.whl#sha256=1480f46b08e60d5fc895996cda1e7d98ead282b7ea49cb95665a2f654e3e264d,pytest-xdist==3.5.0,python-dateutil==2.9.0.post0,requests==2.31.0,rich==13.7.1,setuptools==69.2.0,six==1.16.0,sqlparse==0.4.4,sympy==1.12,tensorboard==2.16.2,tensorboard-data-server==0.7.2,tensorflow==2.16.1,termcolor==2.4.0,torch==2.2.1,typing_extensions==4.10.0,tzdata==2024.1,urllib3==2.2.1,Werkzeug==3.0.1,wheel==0.43.0,wrapt==1.16.0
py312: commands[0]> python -W error::ResourceWarning -W error::DeprecationWarning -W error::PendingDeprecationWarning -W 'ignore:pkg_resources is deprecated as an API:DeprecationWarning' -W 'ignore:datetime.datetime.utcfromtimestamp() is deprecated:DeprecationWarning' -m coverage run -m pytest -p no:randomly tests
Fatal Python error: Segmentation fault
which tensorflow-io-gcs-filesystem package is missing. I guess there might be some compatibility issue between python and package. I will inquire about this issue in community and try to figure it out. Thanks for your help!
Huh, interesting. It would be surprising though if the missing package caused the segmentation fault. A segfault is when code reaches for memory outside of its allotted area.