Python 3.8 support

[shacker]

I know that Django doesn't officially support Python 3.8 yet, but I tried an upgrade anyway. Django seems to work fine and our project is happy, but qcluster crashes on start with:

TypeError: cannot pickle '_thread.lock' object

Seems to be related to multiprocessing changes in py38. As far as I can tell at this early stage, it only seems to crash on MacOS, while Linux is fine (our Linux developers had no problem and our server deploys were fine, but MacOS devs all experienced this crash).

Discussion of this started in #313, but this is a separate issue.

[Solanar]

This is tied to the multiprocessing start method, when it's spawn it can't pickle locks. This shouldn't be unique to python 3.8, have you tried MacOS on < 3.8?

You can try out my fork in #406 to see if it works.

[aaronoppenheimer]

I see this is closed but I still have the issue (python 3.8, django 3.0.3, django-q 1.2.0) - what to do?

[Koed00]

I don't have MacOs available to me anymore. All I can do is reopen the issue and hopefully someone will want to make the effort to address this.

[Koed00]

Moving thread lock issues to #424

[roharvey]

As @Solanar said, this is related to the start method. A very simple workaround is to include this in startup (windows or OS X, but should be a no-op in linux):

from multiprocessing import set_start_method
set_start_method("fork")

The reason this shows up in python3.8 is because the default multiprocessing method changed: https://docs.python.org/3/whatsnew/3.8.html#multiprocessing

[shacker]

Thanks @roharvey . I can confirm that adding this in my django settings allows django-q to work with python3.8 on Mac OS, thank you!

I guess the question for django-q is whether this should just be a documentation suggestion or can it be utilized in a code fix so the workaround isn't needed.

[jeroenbrouwer]

As @Solanar said, this is related to the start method. A very simple workaround is to include this in startup (windows or OS X, but should be a no-op in linux):

from multiprocessing import set_start_method
set_start_method("fork")

The reason this shows up in python3.8 is because the default multiprocessing method changed: https://docs.python.org/3/whatsnew/3.8.html#multiprocessing

Thanks for the suggestion! I've added the following snippet to my manage.py and it works :-)

import platform
if platform.system() != "Linux":
    from multiprocessing import set_start_method
    set_start_method("fork")

[soumojit]

Just came across this thread while facing the same issue although my project is unrelated to django-q.

The same behavior is observed on

Mac OS Big Sur
Python 3.9.0, 3.9.1, 3.9.2

Thanks @Solanar for your solution, saved my day!

[psxpa3]

Hi @Solanar , I am also facing this issue with Python 3.8 on MacOs Monterey. Multi-processing was working fine with Python 3.6. The code snippet is

{code} p = multiprocessing.cpu_count() print("layers, pppp ", self._layers, p, type(p)) with Pool(p) as prs: prs.map(fnction(), [layer1, layer2]) {code}.

Could you please elaborate where the mentioned lines has to be written?. I have setup.py file in my python project, but do not see any manage.py file. Thank you very much in advance