rasterio icon indicating copy to clipboard operation
rasterio copied to clipboard

test_reproject_error_propagation fails on Python 3.14

Open QuLogic opened this issue 10 months ago • 1 comments

Expected behavior and actual behavior.

Python 3.14b1 has changed the default multiprocessing start method from fork to forkserver: https://docs.python.org/dev/whatsnew/3.14.html#multiprocessing

This appears to break the test setup which can't be pickled:

______________ ERROR at setup of test_reproject_error_propagation ______________

data = local('/tmp/pytest-of-mockbuild/pytest-0/test_reproject_error_propagati0')

    @pytest.mark.network
    @pytest.fixture
    def http_error_server(data):
        """Serves files from the test data directory, poorly."""
        import functools
        import multiprocessing
        import http.server
    
        Handler = functools.partial(RangeRequestErrorHandler, directory=str(data))
        httpd = http.server.HTTPServer(("", 0), Handler)
        p = multiprocessing.Process(target=httpd.serve_forever)
>       p.start()

tests/test_warp.py:2126: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.14/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/usr/lib64/python3.14/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/usr/lib64/python3.14/multiprocessing/context.py:300: in _Popen
    return Popen(process_obj)
/usr/lib64/python3.14/multiprocessing/popen_forkserver.py:35: in __init__
    super().__init__(process_obj)
/usr/lib64/python3.14/multiprocessing/popen_fork.py:20: in __init__
    self._launch(process_obj)
/usr/lib64/python3.14/multiprocessing/popen_forkserver.py:47: in _launch
    reduction.dump(process_obj, buf)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <Process name='Process-9' parent=2132 initial>
file = <_io.BytesIO object at 0x7f384f02fa60>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       TypeError: cannot pickle '_thread.lock' object
E       when serializing dict item '_lock'
E       when serializing threading.Condition state
E       when serializing threading.Condition object
E       when serializing dict item '_cond'
E       when serializing threading.Event state
E       when serializing threading.Event object
E       when serializing dict item '_BaseServer__is_shut_down'
E       when serializing http.server.HTTPServer state
E       when serializing http.server.HTTPServer object
E       when serializing tuple item 0
E       when serializing method reconstructor arguments
E       when serializing method object
E       when serializing dict item '_target'
E       when serializing multiprocessing.context.Process state
E       when serializing multiprocessing.context.Process object

I guess maybe the start method should explicitly be set to fork?

Environment Information

  • rasterio version (python -c "import rasterio; print(rasterio.__version__)") 1.4.3
  • GDAL version (python -c "import rasterio; print(rasterio.__gdal_version__)") n/a
  • Python version (python -c "import sys; print(sys.version.replace('\n', ' '))") 3.14.0b1
  • Operation System Information (python -c "import platform; print(platform.platform())") Fedora Rawhide

Installation Method

From sdist.

QuLogic avatar Jun 04 '25 05:06 QuLogic

Thanks for the report @QuLogic !

sgillies avatar Jun 04 '25 13:06 sgillies