pythontex icon indicating copy to clipboard operation
pythontex copied to clipboard

Support for platforms where multiprocessing does not work

Open Grimler91 opened this issue 4 years ago • 0 comments

Hi,

Currently trying to run pythontex on android gives an error due to multiprocessing not working. There's been plenty of discussion on how to deal with this in other python projects, see for example https://github.com/pypa/pip/issues/8161.

With this PR we check if multiprocessing is supported and else set multiprocessing=False. When later creating list of tasks to run we then choose to either run pool.apply_async(function), or to just eval(function).

Error message when currently running pythontex
This is PythonTeX 0.17
Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/synchronize.py", line 28, in <module>
    from _multiprocessing import SemLock, sem_unlink
ImportError: cannot import name 'SemLock' from '_multiprocessing' (/data/data/com.termux/files/usr/lib/python3.8/lib-dynload/_multiprocessing.cpython-38.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/bin/texlive/pythontex", line 62, in <module>
    pythontex.main()
  File "/data/data/com.termux/files/usr/share/texlive/texmf-dist/scripts/pythontex/pythontex3.py", line 2811, in main
    do_multiprocessing(data, temp_data, old_data, engine_dict)
  File "/data/data/com.termux/files/usr/share/texlive/texmf-dist/scripts/pythontex/pythontex3.py", line 1269, in do_multiprocessing
    pool = multiprocessing.Pool(jobs)
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/pool.py", line 191, in __init__
    self._setup_queues()
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/pool.py", line 343, in _setup_queues
    self._inqueue = self._ctx.SimpleQueue()
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/context.py", line 113, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/queues.py", line 336, in __init__
    self._rlock = ctx.Lock()
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/context.py", line 67, in Lock
    from .synchronize import Lock
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/synchronize.py", line 30, in <module>
    raise ImportError("This platform lacks a functioning sem_open" +
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.

I have not updated pythontex2.py, mainly due to that 99 % of all users running into this problem will be using the terminal emulator termux (as I am) where python3 is already the default python version.

Let me know if you are not happy with the code format, associated comments or if I should fix something else in the commits!

Grimler91 avatar Sep 17 '20 09:09 Grimler91