Pickle error when running tests for ChannelsLiveServerTestCase for Windows

[ederic-oytas]

I was following a tutorial, and I've been getting AttributeError: Can't pickle local object 'DaphneProcess.__init__.<locals>.<lambda>'.

I've done further research into this and it turns out to be a problem with Python's multiprocessing library with lambdas on Windows. There's a solution on Stack Overflow but changing code in the standard library is not very maintainable.

I went back to the testing code and stripped away all the test cases and just put one test case. I think this is a problem during the server setup. Here's the code:

from channels.testing import ChannelsLiveServerTestCase


class ChatTests(ChannelsLiveServerTestCase):
    serve_static = True

    def test_1(self):
        self.assertTrue(True)

Are there any other alternatives or solutions to this?

Traceback:

ERROR: test_1 (chat.tests.ChatTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\ederi\Documents\projects\django_chat_project\venv\lib\site-packages\django\test\testcases.py", line 270, in _setup_and_call
    self._pre_setup()
  File "C:\Users\ederi\Documents\projects\django_chat_project\venv\lib\site-packages\channels\testing\live.py", line 52, in _pre_setup
    self._server_process.start()
  File "C:\Users\ederi\AppData\Local\Programs\Python\Python37\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
  File "C:\Users\ederi\AppData\Local\Programs\Python\Python37\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\ederi\AppData\Local\Programs\Python\Python37\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Users\ederi\AppData\Local\Programs\Python\Python37\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\ederi\AppData\Local\Programs\Python\Python37\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'DaphneProcess.__init__.<locals>.<lambda>'

----------------------------------------------------------------------
Ran 0 tests in 0.024s

FAILED (errors=1)
Destroying test database for alias 'default'...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\ederi\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\ederi\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

OS: Windows 10 Using: Django 3.1, channels 2.4.0

[carltongibson]

Hi @ederic-oytas. Thanks for the report. Initial thought is Eeek 😬 not much we can do here — but will leave it open to see if we can adjust somehow. (Need a spot of time to investigate.)

For now, I might suggest giving the WSL a go… — I know that's not ideal.

[ghost]

I had just made this on issue on stackoverflow: https://stackoverflow.com/questions/64050771/channels-selenium-testing-cant-pickle-lambda

I'm having the exact same issue but on MacOS Catalina 10.15.6

[deepio]

It seems like this is no small issue, what do you suggest for the time being?

OS: macOS BigSur 11.2.3 Python: 3.9.1 Channels: 3.0.3 Django: 3.2.3

I was following a tutorial, and I've been getting AttributeError: Can't pickle local object 'DaphneProcess.__init__.<locals>.<lambda>'.

I've done further research into this and it turns out to be a problem with Python's multiprocessing library with lambdas on Windows. There's a solution on Stack Overflow but changing code in the standard library is not very maintainable.

I went back to the testing code and stripped away all the test cases and just put one test case. I think this is a problem during the server setup. Here's the code:

from channels.testing import ChannelsLiveServerTestCase


class ChatTests(ChannelsLiveServerTestCase):
    serve_static = True

    def test_1(self):
        self.assertTrue(True)

Traceback (most recent call last):
  File "/Users/deepio/Documents/GitHub/channels_tut/env/lib/python3.9/site-packages/django/test/testcases.py", line 272, in _setup_and_call
    self._pre_setup()
  File "/Users/deepio/Documents/GitHub/channels_tut/env/lib/python3.9/site-packages/channels/testing/live.py", line 52, in _pre_setup
    self._server_process.start()
  File "/Users/deepio/.pyenv/versions/3.9.1/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/deepio/.pyenv/versions/3.9.1/lib/python3.9/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/deepio/.pyenv/versions/3.9.1/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/deepio/.pyenv/versions/3.9.1/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/deepio/.pyenv/versions/3.9.1/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/deepio/.pyenv/versions/3.9.1/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/deepio/.pyenv/versions/3.9.1/lib/python3.9/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'convert_exception_to_response.<locals>.inner'

[MosGeo]

Hitting the same issue here on Windows 11. Tested both Python 3.9 and 3.10.

[carltongibson]

Same issue was reported here: https://github.com/django/asgiref/issues/313

Comment from there:

I'd say it's less of a bug and more an unobvious restriction in what can be done with multiprocessing.

Comment from above:

... it turns out to be a problem with Python's multiprocessing library with lambdas on Windows.

What do you suggest for the time being?

Until someone can sit down and have a good think about this, it's going to be use a linux VM (or WSL or docker) for the moment.

[Eunsub-thumb]

I got the same issue. os: macOS Monterey 12.4 (M1) python: 3.8

[carltongibson]

I've merged #1906 to enforce fork here, which at least lets test cases work on macOS. (We'll need a fuller solution long-term, and for Windows support.)

Note pre-4.0 this requires the development version of daphne as well. See #1898

[carltongibson]

I'm going to close this as a duplicate of #1207 to focus there.