mosquitto
mosquitto copied to clipboard
Published
20 hours ago •
eclipse
eclipse
Podman Rootless Container Issue - failed to write to /proc/self/oom_score_adj: Permission denied
I am getting an error
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied
When trying to run the latest image of eclipse-mosquitto (e.g. from docker.io).
I originally thought this was a podman bug -> https://github.com/containers/podman/issues/20886
However, after playing around a bit with mosquitto and trying to build using the Dockerfile, maybe the issue is within eclipse-mosquitto itself.
I tried to build using the following command
cd ~/build/mosquitto/git-eclipse-mosquitto/docker/$version
podman build --tag homelab:eclipse-mosquitto -f ./Dockerfile .
I tried:
- 2.0 -> this results in
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied - 2.0-openssl -> this results in
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied - 1.6-openssl -> this works (container is running since 12 minutes now)
I tried a quick diff between the 2.0-openssl and 1.6-openssl Dockerfile
podman@Rock5B-01:~/build/mosquitto/git-eclipse-mosquitto/docker$ diff 2.0-openssl/Dockerfile 1.6-openssl/Dockerfile
6,7c6,7
< ENV VERSION=2.0.18 \
< DOWNLOAD_SHA256=d665fe7d0032881b1371a47f34169ee4edab67903b2cd2b4c083822823f4448a \
---
> ENV VERSION=1.6.15 \
> DOWNLOAD_SHA256=5ff2271512f745bf1a451072cd3768a5daed71e90c5179fae12b049d6c02aa0f \
16d15
< cjson-dev \
64c63
< CFLAGS="-Wall -O2 -I/build/lws/include -I/build" \
---
> CFLAGS="-Wall -O2 -I/build/lws/include" \
83,85c82
< install -s -m755 /build/mosq/apps/mosquitto_ctrl/mosquitto_ctrl /usr/bin/mosquitto_ctrl && \
< install -s -m755 /build/mosq/apps/mosquitto_passwd/mosquitto_passwd /usr/bin/mosquitto_passwd && \
< install -s -m755 /build/mosq/plugins/dynamic-security/mosquitto_dynamic_security.so /usr/lib/mosquitto_dynamic_security.so && \
---
> install -s -m755 /build/mosq/src/mosquitto_passwd /usr/bin/mosquitto_passwd && \
88c85
< install -Dm644 /build/mosq/epl-v20 /usr/share/licenses/mosquitto/epl-v20 && \
---
> install -Dm644 /build/mosq/epl-v10 /usr/share/licenses/mosquitto/epl-v10 && \
92,93c89
< ca-certificates \
< cjson && \
---
> ca-certificates && \
100c96
< COPY docker-entrypoint.sh mosquitto-no-auth.conf /
---
> COPY docker-entrypoint.sh /
There doesn't seem to be a whole lot of differences. Any idea what could be causing the error ?
Uhm, the error message is totally misleading when using Mosquitto 2.x.
Looking at the contents of /home/podman/log/mosquitto01/mosquitto.log seem to reveal the real culprit
(...)
1701606791: Error: Unable to load server certificate "/mosquitto/ssl/mosquitto01.cert". Check certfile.
1701606791: OpenSSL Error[0]: error:02FFF002:system library:func(4095):No such file or directory
1701606791: OpenSSL Error[1]: error:20FFF002:BIO routines:CRYPTO_internal:system lib
1701606791: OpenSSL Error[2]: error:14FFF002:SSL routines:(UNKNOWN)SSL_internal:system lib
1701606836: mosquitto version 2.0.18 starting
1701606836: Config loaded from /mosquitto/config/mosquitto.conf.
1701606836: Opening ipv4 listen socket on port 1883.
1701606836: Opening ipv6 listen socket on port 1883.
1701606836: Opening ipv4 listen socket on port 8885.
1701606836: Opening ipv6 listen socket on port 8885.
1701606836: Error: Unable to load server certificate "/mosquitto/ssl/mosquitto01.cert". Check certfile.
1701606836: OpenSSL Error[0]: error:80000002:system library::No such file or directory
1701606836: OpenSSL Error[1]: error:10080002:BIO routines::system lib
1701606836: OpenSSL Error[2]: error:0A080002:SSL routines::system lib
1701606844: mosquitto version 2.0.18 starting
1701606844: Config loaded from /mosquitto/config/mosquitto.conf.
1701606844: Opening ipv4 listen socket on port 1883.
1701606844: Opening ipv6 listen socket on port 1883.
1701606844: Opening ipv4 listen socket on port 8885.
1701606844: Opening ipv6 listen socket on port 8885.
1701606844: Error: Unable to load server certificate "/mosquitto/ssl/mosquitto01.cert". Check certfile.
1701606844: OpenSSL Error[0]: error:80000002:system library::No such file or directory
1701606844: OpenSSL Error[1]: error:10080002:BIO routines::system lib
1701606844: OpenSSL Error[2]: error:0A080002:SSL routines::system lib
1701607264: mosquitto version 1.6.15 starting
1701607264: Config loaded from /mosquitto/config/mosquitto.conf.
1701607264: Error: Invalid password hash for user admin, removing entry.
1701607264: Opening ipv4 listen socket on port 1883.
1701607264: Opening ipv6 listen socket on port 1883.
1701607264: Opening ipv4 listen socket on port 8885.
1701607264: Opening ipv6 listen socket on port 8885.
1701607264: mosquitto version 1.6.15 running
1701609064: Saving in-memory database to /mosquitto/data/mosquitto.db.
1701610865: Saving in-memory database to /mosquitto/data/mosquitto.db.
1701611515: mosquitto version 1.6.15 terminating
1701611515: Saving in-memory database to /mosquitto/data/mosquitto.db.
1701611575: mosquitto version 2.0.18 starting
1701611575: Config loaded from /mosquitto/config/mosquitto.conf.
1701611575: Opening ipv4 listen socket on port 1883.
1701611575: Opening ipv6 listen socket on port 1883.
1701611575: Opening ipv4 listen socket on port 8885.
1701611575: Opening ipv6 listen socket on port 8885.
1701611575: Error: Unable to load CA certificates. Check cafile "/mosquitto/ssl/ca/ca.pem".
1701611575: Error: Unable to load server certificate "/mosquitto/ssl/server/server.crt". Check certfile.
1701611575: OpenSSL Error[0]: error:02FFF002:system library:func(4095):No such file or directory
1701611575: OpenSSL Error[1]: error:20FFF080:BIO routines:CRYPTO_internal:no such file
1701611575: OpenSSL Error[2]: error:0BFFF002:x509 certificate routines:CRYPTO_internal:system lib
1701611575: OpenSSL Error[3]: error:09FFF06C:PEM routines:CRYPTO_internal:no start line
1701611575: OpenSSL Error[4]: error:14FFF009:SSL routines:(UNKNOWN)SSL_internal:PEM lib
1701611595: mosquitto version 2.0.18 starting
1701611595: Config loaded from /mosquitto/config/mosquitto.conf.
1701611595: Opening ipv4 listen socket on port 1883.
1701611595: Opening ipv6 listen socket on port 1883.
1701611595: Opening ipv4 listen socket on port 8885.
1701611595: Opening ipv6 listen socket on port 8885.
1701611595: Error: Unable to load CA certificates. Check cafile "/mosquitto/ssl/ca/ca.pem".
1701611595: Error: Unable to load server certificate "/mosquitto/ssl/server/server.crt". Check certfile.
1701611595: OpenSSL Error[0]: error:02FFF002:system library:func(4095):No such file or directory
1701611595: OpenSSL Error[1]: error:20FFF080:BIO routines:CRYPTO_internal:no such file
1701611595: OpenSSL Error[2]: error:0BFFF002:x509 certificate routines:CRYPTO_internal:system lib
1701611595: OpenSSL Error[3]: error:09FFF06C:PEM routines:CRYPTO_internal:no start line
1701611595: OpenSSL Error[4]: error:14FFF009:SSL routines:(UNKNOWN)SSL_internal:PEM lib
My mosquitto.conf
# MQTT
listener 1883
protocol mqtt
persistence true
persistence_location /mosquitto/data/
log_dest file /mosquitto/log/mosquitto.log
# MQTTS
listener 8885
cafile /mosquitto/ssl/ca/ca.pem
certfile /mosquitto/ssl/server/server.crt
keyfile /mosquitto/ssl/server/server.key
tls_version tlsv1.2
## Authentication ##
# By default, Mosquitto >=2.0 allows only authenticated connections. Change to true to enable anonymous connections.
allow_anonymous false
password_file /mosquitto/config/password.txt
And my podman / Docker compose.yml file is
version: '3'
networks:
podman:
services:
mosquitto01:
# image: localhost/homelab:eclipse-mosquitto
image: eclipse-mosquitto
container_name: mosquitto01
volumes:
- ~/config/containers/mosquitto01:/mosquitto/config
- ~/data/mosquitto01:/mosquitto/data
- ~/log/mosquitto01:/mosquitto/log
- ~/certificates/mosquitto01:/mosquitto/ssl
ports:
- 1883:1883
- 8885:8885
- 9001:9001
networks:
- podman
capabilities: {CAP_NET_RAW,CAP_NET_BIND_SERVICE}
Am I using an unsupported cypher / library ? Is the openssl version I used too recent for Mosquitto ?
OpenSSL 3.0.11 19 Sep 2023 (Library: OpenSSL 3.0.11 19 Sep 2023)
It's also generating this error during certificate creation
20C02385FFFF0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:../crypto/evp/evp_fetch.c:373:Global default library context, Algorithm (nodes : 0), Properties (<null>)
20C02385FFFF0000:error:03000086:digital envelope routines:do_sigver_init:initialization error:../crypto/evp/m_sigver.c:253:
And server.crt is empty :cry:.
I tried to generate it using
openssl x509 -provider legacy -provider default -nodes -req -in "${basefolder}/server/server.csr" -CA "${basefolder}/ca/ca.crt" -CAkey "${basefolder}/ca/ca.key" -CAcreateserial -out "${basefolder}/server/server.crt" -days $duration
Legacy procider should've make it work. I also added -provider default after -provider legacy but not difference.
Actually the issue might be related to the "-nodes" option I used here (in some cases it's required, otherwise the prompt Enter the PEM Passphrase appears).
Updated to this generates the server.crt certificate correctly
openssl x509 -provider legacy -provider default -req -in "${basefolder}/server/server.csr" -CA "${basefolder}/ca/ca.crt" -CAkey "${basefolder}/ca/ca.key" -CAcreateserial -out "${basefolder}/server/server.crt" -days $duration
Onto the next error
1701612073: mosquitto version 2.0.18 running
1701612376: mosquitto version 2.0.18 terminating
1701612376: Saving in-memory database to /mosquitto/data//mosquitto.db.
1701612386: mosquitto version 2.0.18 starting
1701612386: Config loaded from /mosquitto/config/mosquitto.conf.
1701612386: Opening ipv4 listen socket on port 1883.
1701612386: Opening ipv6 listen socket on port 1883.
1701612386: Opening ipv4 listen socket on port 8885.
1701612386: Opening ipv6 listen socket on port 8885.
1701612386: Error: Unable to load CA certificates. Check cafile "/mosquitto/ssl/ca/ca.pem".
1701612386: Error: Unable to load server certificate "/mosquitto/ssl/server/server.crt". Check certfile.
1701612386: OpenSSL Error[0]: error:02FFF002:system library:func(4095):No such file or directory
1701612386: OpenSSL Error[1]: error:20FFF080:BIO routines:CRYPTO_internal:no such file
1701612386: OpenSSL Error[2]: error:0BFFF002:x509 certificate routines:CRYPTO_internal:system lib
1701612386: OpenSSL Error[3]: error:09FFF06C:PEM routines:CRYPTO_internal:no start line
1701612386: OpenSSL Error[4]: error:14FFF009:SSL routines:(UNKNOWN)SSL_internal:PEM lib
1701612442: mosquitto version 2.0.18 starting
1701612442: Config loaded from /mosquitto/config/mosquitto.conf.
1701612442: Opening ipv4 listen socket on port 1883.
1701612442: Opening ipv6 listen socket on port 1883.
1701612442: Opening ipv4 listen socket on port 8885.
1701612442: Opening ipv6 listen socket on port 8885.
1701612442: Error: Unable to load server certificate "/mosquitto/ssl/server/server.crt". Check certfile.
1701612442: OpenSSL Error[0]: error:09FFF06C:PEM routines:CRYPTO_internal:no start line
1701612442: OpenSSL Error[1]: error:14FFF009:SSL routines:(UNKNOWN)SSL_internal:PEM lib
1701613159: mosquitto version 2.0.18 starting
1701613159: Config loaded from /mosquitto/config/mosquitto.conf.
1701613159: Opening ipv4 listen socket on port 1883.
1701613159: Opening ipv6 listen socket on port 1883.
1701613159: Opening ipv4 listen socket on port 8885.
1701613159: Opening ipv6 listen socket on port 8885.
1701613159: Error: Unable to load server key file "/mosquitto/ssl/server/server.key". Check keyfile.
1701613159: OpenSSL Error[0]: error:0BFFF074:x509 certificate routines:CRYPTO_internal:extension value error
Probably some permission errors. But what worries me is the last line
OpenSSL Error[0]: error:0BFFF074:x509 certificate routines:CRYPTO_internal:extension value error
Now it works but I do not know why ...
I regenerated all certificates, keys and CA ...
Then from outside the container I did
chown -R $user:$user "${basefolder}"
chmod 0755 "${basefolder}"
chmod 0755 "${basefolder}/ca"
chmod 0755 "${basefolder}/client"
chmod 0755 "${basefolder}/server"
chmod 0640 ${basefolder}/ca/*
chmod 0640 ${basefolder}/client/*
chmod 0640 ${basefolder}/server/*
And now it seems to work
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b9247e4d1504 docker.io/library/eclipse-mosquitto:latest /usr/sbin/mosquit... 51 minutes ago Up 2 minutes 0.0.0.0:1883->1883/tcp, 0.0.0.0:8885->8885/tcp, 0.0.0.0:9001->9001/tcp mosquitto01