Fix Docker for pyzmq

[s-martin]

g++ was missing to create pyzmq.

This happened to me under Windows

[s-martin]

Building the container works now, but #1838 still doesn't work.

[AlvinSchiller]

OK the build works, but staring throws an error:

mpd exited with code 0
mpd      | exception: Failed to open '/home/pi/.config/mpd/mpd.conf': No such file or directory
[...]
jukebox  | run_jukebox.py: error: argument -c/--conf: can't open '/home/pi/RPi-Jukebox-RFID/shared/settings/jukebox.yaml': [Errno 2] No such file or directory: '/home/pi/RPi-Jukebox-RFID/shared/settings/jukebox.yaml'

Did you copy the yaml? docker.md#prerequisites

[Groovylein]

OK the build works, but staring throws an error:

mpd exited with code 0
mpd      | exception: Failed to open '/home/pi/.config/mpd/mpd.conf': No such file or directory
[...]
jukebox  | run_jukebox.py: error: argument -c/--conf: can't open '/home/pi/RPi-Jukebox-RFID/shared/settings/jukebox.yaml': [Errno 2] No such file or directory: '/home/pi/RPi-Jukebox-RFID/shared/settings/jukebox.yaml'

Did you copy the yaml? docker.md#prerequisites

Yes, everything is there.

Changing these volume paths to /home/pi let's the jukebox start: https://github.com/s-martin/RPi-Jukebox-RFID/blob/42c2e8db97d98599e153cb76763edd3891e5368d/docker/docker-compose.yml#L42

The same applies for mpd: https://github.com/s-martin/RPi-Jukebox-RFID/blob/42c2e8db97d98599e153cb76763edd3891e5368d/docker/docker-compose.yml#L17

[AlvinSchiller]

The same applies for mpd: https://github.com/s-martin/RPi-Jukebox-RFID/blob/42c2e8db97d98599e153cb76763edd3891e5368d/docker/docker-compose.yml#L17

Shouldn't they be overwritten from *linux.yml (but i really know docker compose)? Looks like- ../src/jukebox:xxx is missing here?

[s-martin]

Reference: https://github.com/zeromq/pyzmq/wiki/Building-and-Installing-PyZMQ

[s-martin]

I got the webapp running under a Github Codespace using Ubuntu. So I guess my issues on Windows are Windows-related

[s-martin]

Todo: check pyzmq and libzmq version on a working raspberry instance

[s-martin]

@Groovylein can you check again? works for me now under WSL

[s-martin]

#2050

[votti]

I had the pyzmq issues building on linux as well (future3/develop), but these changes seem to work. Cheers!

[Groovylein]

@Groovylein can you check again? works for me now under WSL

Oh missed that. Building works with the changes.

[Groovylein]

My problem now is that I still get the errors:

jukebox  | 22.01.2024 20:32:31 -   67:server.py          - jb.rpc.server        - MainThread      - INFO     - Init RPC Server (Pyzmq version: 21.0.2; ZMQ version: 4.3.3; has draft API: False)
jukebox  | 22.01.2024 20:32:31 -   75:server.py          - jb.rpc.server        - MainThread      - DEBUG    - Connected to address 'inproc://JukeBoxRpcServer'
jukebox  | 22.01.2024 20:32:31 -   81:server.py          - jb.rpc.server        - MainThread      - DEBUG    - Connected to address 'tcp://*:5555'
jukebox  | Traceback (most recent call last):
jukebox  |   File "/root/RPi-Jukebox-RFID/src/jukebox/run_jukebox.py", line 67, in <module>
jukebox  |     main()
jukebox  |   File "/root/RPi-Jukebox-RFID/src/jukebox/run_jukebox.py", line 63, in main
jukebox  |     myjukebox.run()
jukebox  |   File "/root/RPi-Jukebox-RFID/src/jukebox/jukebox/daemon.py", line 227, in run
jukebox  |     self.rpc_server = RpcServer()
jukebox  |   File "/root/RPi-Jukebox-RFID/src/jukebox/jukebox/rpc/server.py", line 86, in __init__
jukebox  |     self.socket.bind(websocket_address)
jukebox  |   File "/root/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/zmq/sugar/socket.py", line 172, in bind
jukebox  |     super().bind(addr)
jukebox  |   File "zmq/backend/cython/socket.pyx", line 540, in zmq.backend.cython.socket.Socket.bind
jukebox  |   File "zmq/backend/cython/checkrc.pxd", line 28, in zmq.backend.cython.checkrc._check_rc
jukebox  | zmq.error.ZMQError: Protocol not supported
jukebox  | 22.01.2024 20:32:31 -   31:daemon.py          - jb.daemon            - MainThread      - DEBUG    - Active Threads = [<_MainThread(MainThread, stopped 139642232436544)>, <Thread(PlayerEventLoop, started daemon 139642181654272)>, <MultiTimer(host.timer.cputemp, started daemon 139642154718976)>]
jukebox exited with code 0

[Groovylein]

When having one line: RUN export ZMQ_PREFIX=${PREFIX} && export ZMQ_DRAFT_API=1 && pip install -v --no-binary pyzmq --pre pyzmq

I get another error:

jukebox  | 22.01.2024 21:27:48 -   67:server.py          - jb.rpc.server        - MainThread      - INFO     - Init RPC Server (Pyzmq version: 25.1.2; ZMQ version: 4.3.4; has draft API: True)
jukebox  | 22.01.2024 21:27:48 -   75:server.py          - jb.rpc.server        - MainThread      - DEBUG    - Connected to address 'inproc://JukeBoxRpcServer'
jukebox  | 22.01.2024 21:27:48 -   81:server.py          - jb.rpc.server        - MainThread      - DEBUG    - Connected to address 'tcp://*:5555'
jukebox  | Traceback (most recent call last):
jukebox  |   File "/root/RPi-Jukebox-RFID/src/jukebox/run_jukebox.py", line 67, in <module>
jukebox  |     main()
jukebox  |   File "/root/RPi-Jukebox-RFID/src/jukebox/run_jukebox.py", line 63, in main
jukebox  |     myjukebox.run()
jukebox  |   File "/root/RPi-Jukebox-RFID/src/jukebox/jukebox/daemon.py", line 227, in run
jukebox  |     self.rpc_server = RpcServer()
jukebox  |   File "/root/RPi-Jukebox-RFID/src/jukebox/jukebox/rpc/server.py", line 86, in __init__
jukebox  |     self.socket.bind(websocket_address)
jukebox  |   File "/root/RPi-Jukebox-RFID/.venv/lib/python3.9/site-packages/zmq/sugar/socket.py", line 302, in bind
jukebox  |     super().bind(addr)
jukebox  |   File "zmq/backend/cython/socket.pyx", line 564, in zmq.backend.cython.socket.Socket.bind
jukebox  |   File "zmq/backend/cython/checkrc.pxd", line 28, in zmq.backend.cython.checkrc._check_rc
jukebox  | zmq.error.ZMQError: Protocol not supported (addr='ws://*:5556')
jukebox  | 22.01.2024 21:27:48 -   31:daemon.py          - jb.daemon            - MainThread      - DEBUG    - Active Threads = [<_MainThread(MainThread, stopped 140644536080192)>, <Thread(PlayerEventLoop, started daemon 140644485297920)>, <MultiTimer(host.timer.cputemp, started daemon 140644391253760)>]
jukebox exited with code 1

[s-martin]

#2268 may be related

[s-martin]

Lots of comments here now. :-D But with the latest changes it seems to work on Windows itself and Linux (WSL Ubuntu).

@AlvinSchiller @pabera @Groovylein if anyone of you could also check, would be nice.

[AlvinSchiller]

~~Did we change anything for this part? Or is it maybe also the prerelease of pyzmq which makes a change? Could you test it without the --pre?~~

Nevermind, didn't see the last commit

[pabera]

This is not entirely fixed. On my Windows machine, it even doesn't work with the updated pip command. Reason for this is that the downloaded, pre-compiled libzmq does not work on my system (x86_64), even though I used the same machine to compile it before. When I compile libzmq manually in the Dockerfile, it works.

This means, we need to enable a way to build libzmq locally in order to use it in case the pre-built versions do not work.

I am working on that.

[AlvinSchiller]

Could we use the bundled libzmq for that (ZMQ_PREFIX="bundled") ? No further setup of libzmq would be needed if it works.

[pabera]

Not sure. In my environment, the problem is that pyzmq (actually g++) cannot compile the lib properly and fails. It then falls back on the bundled, none-draft version that pyzmq is shipped with.

[AlvinSchiller]

Ah ok. Maybe settings the libzmq version explicitly could help in a first step -> export PYZMQ_LIBZMQ_VERSION=4.3.5

Or turn off the fallback? -> export PYZMQ_NO_BUNDLE=ON

[AlvinSchiller]

Might be the cause for the problems here aswell? #2302

[pabera]

Pre-building the libzmq does not work well as many users have different host architectures and OS. The goal should be to build an individual Docker image that just builds libzmq for the host it's running on (1-time build) Once built, it can be used in docker-compose instead of downloading the pre-built version from our repos.

I am working on this in a branch, but haven't had the time to bring it to a draft.

[notapirate]

In commit 1ae54c7b15b56a5595c56aca0ce449e3b5fc6054 changes where made to adjust docker/docker-compose.linux.yaml to docker/docker-compose.mac.yaml. IMHO this doesn't work because other than on mac or windows, on a linux host the already running pulseaudio is used, but for that to work, the UID within the docker container must match the UID of the user outside the container, which doesn't work anymore with 1ae54c7b15b56a5595c56aca0ce449e3b5fc6054, so I vote to undo that commit.

My docker tests show that based on current branch future3/develop, 2 things need to be changed to make it work again:

1. In jukebox.Dockerfile move the commands to untar libzmq up to the area before the User is changed, because on a linux-based host this user is limited and can't write to /usr/local.
2. As @pabera already noted on Feb 20 (I created a separate issue, #2302), the prebuilt libzmq doesn't work on x86_64 on bullseye. For testing, one can use the debian:bookworm-slim image for jukebox (netcat package also needs to change to netcat-openbsd) and see if pyzmq works. This might cause other problems (see #2295 for a summary of bookworm experiences), but it should show if the pyzmq problem is solved then.

On my local build I did both changes and it works fine.

[s-martin]

Will check this :)

[pabera]

@s-martin Could you check if the pyzmq problem is solved for you too? This issue has been the reason I provided #2307 :-)

[s-martin]

#2307 provides a working solution