Python 3.13.0 beta 1 crashes

[kurtmckee]

Describe the bug

When pyfakefs is loaded via the pytest plugin on Python 3.13.0 beta 1, I'm seeing crashes.

[!Note]

I've confirmed this behavior with pyfakefs 5.5.0, 5.4.1, and 5.3.0, and with pytest 8.2.0 and 8.0.0. However, this does not happen with Python 3.12.

It appears that this is specific to Python 3.13.0 beta 1.

========================================================= test session starts =========================================================
platform linux -- Python 3.13.0b1, pytest-8.2.0, pluggy-1.5.0
rootdir: /home/kurt/dev/test
configfile: pyproject.toml
plugins: pyfakefs-5.5.0
collected 1 item                                                                                                                      

test_demo.py E                                                                                                                  [100%]Traceback (most recent call last):
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/pyfakefs/fake_filesystem.py", line 1678, in get_object_from_normpath
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/pyfakefs/fake_file.py", line 562, in get_entry
KeyError: 'home'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/_pytest/config/__init__.py", line 178, in main
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/pluggy/_hooks.py", line 513, in __call__
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/pluggy/_manager.py", line 120, in _hookexec
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/pluggy/_callers.py", line 103, in _multicall
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/_pytest/main.py", line 332, in pytest_cmdline_main
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/_pytest/main.py", line 317, in wrap_session
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/pyfakefs/fake_os.py", line 1426, in wrapped
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/pyfakefs/fake_os.py", line 456, in chdir
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/pyfakefs/fake_filesystem.py", line 2940, in confirmdir
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/pyfakefs/fake_filesystem.py", line 1760, in resolve
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/pyfakefs/fake_filesystem.py", line 1689, in get_object_from_normpath
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/pyfakefs/fake_filesystem.py", line 430, in raise_os_error
FileNotFoundError: [Errno 2] No such file or directory in the fake filesystem: '/home/kurt/dev/test'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kurt/dev/test/.venv/bin/pytest", line 8, in <module>
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/_pytest/config/__init__.py", line 206, in console_main
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/_pytest/config/__init__.py", line 186, in main
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/_pytest/config/__init__.py", line 1135, in _ensure_unconfigure
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/pluggy/_hooks.py", line 513, in __call__
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/pluggy/_manager.py", line 120, in _hookexec
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/pluggy/_callers.py", line 103, in _multicall
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/_pytest/faulthandler.py", line 45, in pytest_unconfigure
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/pyfakefs/fake_os.py", line 1426, in wrapped
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/pyfakefs/fake_os.py", line 326, in close
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/pyfakefs/fake_filesystem.py", line 885, in get_open_file
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/pyfakefs/fake_filesystem.py", line 907, in get_open_files
  File "/home/kurt/dev/test/.venv/lib/python3.13/site-packages/pyfakefs/fake_filesystem.py", line 430, in raise_os_error
OSError: [Errno 9] Bad file descriptor in the fake filesystem: '9'

How To Reproduce

Create a file named test_demo.py with these contents:

def test_pyfakefs(fs):
    pass

Create a virtual environment using Python 3.13:

$ python3.13 -m venv .venv
$ source .venv/bin/activate
$ pip install pytest pyfakefs
$
$ # At the time of writing, these packages are installed:
$ pip freeze
iniconfig==2.0.0
packaging==24.0
pluggy==1.5.0
pyfakefs==5.5.0
pytest==8.2.0

Then, run this command to see the failure:

pytest test_demo.py

Your environment

With the virtual environment (created using the steps above) activated, here are the OS/Python/pyfakefs/pytest versions installed:

Linux-6.5.0-28-generic-x86_64-with-glibc2.35
Python 3.13.0b1 (main, May  9 2024, 19:41:45) [GCC 11.4.0]
pyfakefs 5.5.0
pytest 8.2.0

[mrbean-bremen]

Thanks! I hadn't tested with 3.13 yet, but it was on my to-do list. Note that official pyfakefs support for 3.13 will not be added before the official Python release, as this is a moving target - for 3.12 they changed the pathlib implementation twice during the alpha/beta phases. I'm going to add tests and preliminary support though, now that the beta 1 is out - hopefully soon.

[kurtmckee]

official pyfakefs support for 3.13 will not be added before the official Python release

That matches my own support ideals. I try to test my projects against Python pre-releases as an early flag for potential issues, and pyfakefs was working just fine two weeks ago on 3.13.0 alpha 6, so this is related to something changed in 3.13.0 beta 1.

Thanks for the quick response! :heart:

[mrbean-bremen]

Just FYI: I'll wait for the beta 2 to get out, as the beta 1 does not work under non-Englisch Windows systems (which I have). This is fixed now, and will be released with the next beta.

[kurtmckee]

Sounds good! Thank you!

[arkamar]

I just run the tests with python 3.13.0 beta2 and this is the result:

================================================= test session starts =================================================
platform linux -- Python 3.13.0b2, pytest-8.2.2, pluggy-1.5.0 -- /var/tmp/portage/dev-python/pyfakefs-5.5.0/work/pyfakefs-5.5.0-python3_13/install/usr/bin/python3.13
cachedir: .pytest_cache
rootdir: /var/tmp/portage/dev-python/pyfakefs-5.5.0/work/pyfakefs-5.5.0
configfile: pyproject.toml
collecting ... collected 2575 items

pyfakefs/pytest_tests/hook_test/pytest_hook_test.py::test_1 Traceback (most recent call last):
  File "/var/tmp/portage/dev-python/pyfakefs-5.5.0/work/pyfakefs-5.5.0/pyfakefs/fake_filesystem.py", line 1678, in get_object_from_normpath
  File "/var/tmp/portage/dev-python/pyfakefs-5.5.0/work/pyfakefs-5.5.0/pyfakefs/fake_file.py", line 562, in get_entry
KeyError: 'usr'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.13/linecache.py", line 100, in updatecache
  File "/var/tmp/portage/dev-python/pyfakefs-5.5.0/work/pyfakefs-5.5.0/pyfakefs/fake_os.py", line 1426, in wrapped
  File "/var/tmp/portage/dev-python/pyfakefs-5.5.0/work/pyfakefs-5.5.0/pyfakefs/fake_os.py", line 693, in stat
  File "/var/tmp/portage/dev-python/pyfakefs-5.5.0/work/pyfakefs-5.5.0/pyfakefs/fake_filesystem.py", line 673, in stat
  File "/var/tmp/portage/dev-python/pyfakefs-5.5.0/work/pyfakefs-5.5.0/pyfakefs/fake_filesystem.py", line 1760, in resolve
  File "/var/tmp/portage/dev-python/pyfakefs-5.5.0/work/pyfakefs-5.5.0/pyfakefs/fake_filesystem.py", line 1689, in get_object_from_normpath
  File "/var/tmp/portage/dev-python/pyfakefs-5.5.0/work/pyfakefs-5.5.0/pyfakefs/fake_filesystem.py", line 430, in raise_os_error
FileNotFoundError: [Errno 2] No such file or directory in the fake filesystem: '/usr/lib/python3.13/linecache.py'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/tmp/portage/dev-python/pyfakefs-5.5.0/work/pyfakefs-5.5.0/pyfakefs/fake_filesystem.py", line 1678, in get_object_from_normpath
  File "/var/tmp/portage/dev-python/pyfakefs-5.5.0/work/pyfakefs-5.5.0/pyfakefs/fake_file.py", line 562, in get_entry
KeyError: 'work'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.13/linecache.py", line 100, in updatecache
  File "/var/tmp/portage/dev-python/pyfakefs-5.5.0/work/pyfakefs-5.5.0/pyfakefs/fake_os.py", line 1426, in wrapped
  File "/var/tmp/portage/dev-python/pyfakefs-5.5.0/work/pyfakefs-5.5.0/pyfakefs/fake_os.py", line 693, in stat
  File "/var/tmp/portage/dev-python/pyfakefs-5.5.0/work/pyfakefs-5.5.0/pyfakefs/fake_filesystem.py", line 673, in stat
  File "/var/tmp/portage/dev-python/pyfakefs-5.5.0/work/pyfakefs-5.5.0/pyfakefs/fake_filesystem.py", line 1760, in resolve
  File "/var/tmp/portage/dev-python/pyfakefs-5.5.0/work/pyfakefs-5.5.0/pyfakefs/fake_filesystem.py", line 1689, in get_object_from_normpath
  File "/var/tmp/portage/dev-python/pyfakefs-5.5.0/work/pyfakefs-5.5.0/pyfakefs/fake_filesystem.py", line 430, in raise_os_error
FileNotFoundError: [Errno 2] No such file or directory in the fake filesystem: '/var/tmp/portage/dev-python/pyfakefs-5.5.0/work/pyfakefs-5.5.0/pyfakefs/fake_open.py'

The log continues with a lot of more tracebacks.

[mrbean-bremen]

Yes, I'm on it, but it may take a bit. I'm still not done with the pathlib changes.

[arkamar]

Great, thanks! I just wanted to share the output if I have it. Maybe also for others to let them know that the issue appears with beta 2 as well.

[mrbean-bremen]

Should work for Beta2 in main now. This is not the final version - there is a workaround for an issue that may be related to a bug in the beta2, and another workaround that will be reworked with #1025, and some probable peformance issue, but I will wait for a release closer to the final release before working on these.

I will probably make a new release after #1025 has been finished and merged.

[mgorny]

Thanks a lot for your work!

[mrbean-bremen]

Just for the curious: the concrete recursion was caused by an optimization in linecache, which now imports os not top-level, but locally. I exclude linecache from patching, as it is used for loading code files, but in this case os is loaded at runtime by linecache, and to check who is loading it, I use traceback to get the stack. traceback itself uses linecache, which caused the recursion.

[kurtmckee]

Thank you so much for your work on this!