[android] posix semaphores unavailable (pipupgrade crashes)

[danieldjewell]

Hello!

Summary

It looks like pipupgrade will not run on Android (at all?) as POSIX semaphores are not enabled (I haven't tested all Android versions/device combinations - it's very possible that on x86 Android devices this might work.)

Because of the lack of POSIX Semaphores, pipupgrade will not run at all - see trace below. The two examples are (1) running on Termux, where their package build of python3 specifically disables POSIX semaphores on build (because, I presume, it's not an available function)... and also of "archlinux running in a proot on Termux for Android" (basically a chroot of Archlinux). This also fails because, ultimately, Semaphores are just not available (the displayed error is different - the Termux python was compiled without POSIX Semaphores and the Arch version was compiled with -- the Arch version tries but gets a permission denied - end result is the same: it doesn't work.)

Thoughts

I haven't had a chance to go through pipupgrade's code fully - however the issue is definitely related to using multiprocessing:

https://github.com/achillesrasquinha/pipupgrade/blob/33c9a0c01ffa2d5f987b1b8ff754553dd9993424/src/pipupgrade/parallel.py#L1-L9

Arguably, Python itself should probably "fail" more elegantly in these cases (e.g. it determines that Semaphores aren't available on the platform it's running on, generates a warning, and then finds another method to run the code ... but then there's a whole argument of doing something in a way that the programmer didn't explicitly ask for...)

Potential Solution

One solution might be for pipupgrade to detect if instantiating a multiprocessing.Pool() object throws an error and to fall back to single-thread operation (without multiprocessing - or to potentially set an option in multiprocessing - however, I'm not 100% sure that there is even an option that can be set to work around this). As of now, using pipupgrade -j 1 to specify a single thread still throws this error because multiprocessing is still used regardless of the number of jobs set.

Related

This might be related to #40 ... not 100% sure. See some of the Python bugs I've linked to below.

Further Reading

Python Issue 3770 - multiprocessing crashes on systems with HAVE_SEM_OPEN=0 Python Issue 33725 - Python crashing on mac with multiprocessing

Platform Info

• Android 9/aarch64 (arm64)
• Python 3.7.5 (Termux) // Python 3.7.4 (Arch On Arm)
• pipupgrade version: 1.5.2 (latest avaialble on PyPi)

Termux

*~ pipupgrade -c -i*
Checking...
Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.7/multiprocessing/synchronize.py", line 28, in <module>
    from _multiprocessing import SemLock, sem_unlink
ImportError: cannot import name 'SemLock' from '_multiprocessing' (/data/data/com.termux/files/usr/lib/python3.7/lib-dynload/_multiprocessing.cpython-37m.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/bin/pipupgrade", line 8, in <module>
    sys.exit(main())
  File "/data/data/com.termux/files/usr/lib/python3.7/site-packages/pipupgrade/cli/__init__.py", line 14, in wrapper
    return fn(**params)
  File "/data/data/com.termux/files/usr/lib/python3.7/site-packages/pipupgrade/commands/__init__.py", line 112, in command
    with parallel.pool(processes = jobs) as pool:
  File "/data/data/com.termux/files/usr/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/data/data/com.termux/files/usr/lib/python3.7/site-packages/pipupgrade/parallel.py", line 7, in pool
    pool = mp.Pool(*args, **kwargs)
  File "/data/data/com.termux/files/usr/lib/python3.7/multiprocessing/context.py", line 119, in Pool
    context=self.get_context())
  File "/data/data/com.termux/files/usr/lib/python3.7/multiprocessing/pool.py", line 158, in __init__
    self._setup_queues()
  File "/data/data/com.termux/files/usr/lib/python3.7/multiprocessing/pool.py", line 251, in _setup_queues
    self._inqueue = self._ctx.SimpleQueue()
  File "/data/data/com.termux/files/usr/lib/python3.7/multiprocessing/context.py", line 112, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
  File "/data/data/com.termux/files/usr/lib/python3.7/multiprocessing/queues.py", line 332, in __init__
    self._rlock = ctx.Lock()
  File "/data/data/com.termux/files/usr/lib/python3.7/multiprocessing/context.py", line 66, in Lock
    from .synchronize import Lock
  File "/data/data/com.termux/files/usr/lib/python3.7/multiprocessing/synchronize.py", line 32, in <module>
    " synchronization primitives needed will not" +
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.

Arch proot on Termux

Traceback (most recent call last):
  File "/bin/pipupgrade", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3.7/site-packages/pipupgrade/cli/__init__.py", line 14, in wrapper
    return fn(**params)
  File "/usr/lib/python3.7/site-packages/pipupgrade/commands/__init__.py", line 112, in command
    with parallel.pool(processes = jobs) as pool:
  File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/usr/lib/python3.7/site-packages/pipupgrade/parallel.py", line 7, in pool
    pool = mp.Pool(*args, **kwargs)
  File "/usr/lib/python3.7/multiprocessing/context.py", line 119, in Pool
    context=self.get_context())
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 158, in __init__
    self._setup_queues()
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 251, in _setup_queues
    self._inqueue = self._ctx.SimpleQueue()
  File "/usr/lib/python3.7/multiprocessing/context.py", line 112, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
  File "/usr/lib/python3.7/multiprocessing/queues.py", line 332, in __init__
    self._rlock = ctx.Lock()
  File "/usr/lib/python3.7/multiprocessing/context.py", line 67, in Lock
    return Lock(ctx=self.get_context())
  File "/usr/lib/python3.7/multiprocessing/synchronize.py", line 162, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/usr/lib/python3.7/multiprocessing/synchronize.py", line 59, in __init__
    unlink_now)
PermissionError: [Errno 13] Permission denied

[danieldjewell]

FWIW, I just checked out tag v1.5.1 (commit e69dbb9) (before the af4baec commit) -- and it seems to work beautifully. Perhaps a fallback option to the old way if multiprocessing.Pool() throws an error?

[achillesrasquinha]

Hi, sorry for this very late response but you're probably right in not using multiprocessing.Pool for a single job.

[Mark-Joy]

Hi @achillesrasquinha, I also got this error on Termux-Android. How to solve it?

λ [~] $ pipupgrade --self
Checking...                                                   Updating pipupgrade...
pipupgrade upto date.                                         λ [~] $ pipupgrade --version
0.2.0
λ [~] $ pipupgrade --check
Checking...

Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.9/multiprocessing/synchronize.py", line 28, in <module>
    from _multiprocessing import SemLock, sem_unlink
ImportError: cannot import name 'SemLock' from '_multiprocessing' (/data/data/com.termux/files/usr/lib/python3.9/lib-dynload/_multiprocessing.cpython-39.so)
                                                              During handling of the above exception, another exception occurred:                                                                                                                       Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pipupgrade/commands/__init__.py", line 79, in command
    return _command(**ARGUMENTS)
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pipupgrade/commands/__init__.py", line 239, in _command
    with parallel.no_daemon_pool(processes = a.jobs) as pool:
  File "/data/data/com.termux/files/usr/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/bpyutils/parallel.py", line 66, in no_daemon_pool
    with pool(class_ = NoDaemonPool, *args, **kwargs) as p:
  File "/data/data/com.termux/files/usr/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/bpyutils/parallel.py", line 60, in pool
    pool = class_(*args, **kwargs)
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/bpyutils/parallel.py", line 21, in __init__
    self.super.__init__(*args, **kwargs)
  File "/data/data/com.termux/files/usr/lib/python3.9/concurrent/futures/process.py", line 642, in __init__
    self._call_queue = _SafeQueue(
  File "/data/data/com.termux/files/usr/lib/python3.9/concurrent/futures/process.py", line 165, in __init__
    super().__init__(max_size, ctx=ctx)
  File "/data/data/com.termux/files/usr/lib/python3.9/multiprocessing/queues.py", line 43, in __init__
    self._rlock = ctx.Lock()
  File "/data/data/com.termux/files/usr/lib/python3.9/multiprocessing/context.py", line 67, in Lock
    from .synchronize import Lock
  File "/data/data/com.termux/files/usr/lib/python3.9/multiprocessing/synchronize.py", line 30, in <module>
    raise ImportError("This platform lacks a functioning sem_open" +
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.

An error occured while performing the above command. This could be an issue with
"pipupgrade". Kindly post an issue at https://github.com/achillesrasquinha/pipupgrade/issues
λ [~] $