self-hosted
self-hosted copied to clipboard
getsentry
Constant dmesg messages regarding docker network interface changes
Self-Hosted Version
22.8.0
CPU Architecture
x86_64
Docker Version
20.10.18, build b40c2f6
Docker Compose Version
2.7.0
Steps to Reproduce
- Get an instance running with a Docker version < 20.10.18
- Update Docker to 20.10.18
- Re-do a
./install.shto make sure all containers are setup correctly - Check
dmesg -cregularely (like once a minute)
I am running Debian 11
Expected Result
No permanent dmesg messages regarding network changes
Actual Result
I get an email every minute regarding changes inside dmesg which currently is always:
[ 1026.442192] br-888cec1778ee: port 11(veth12f3420) entered blocking state
[ 1026.442195] br-888cec1778ee: port 11(veth12f3420) entered disabled state
[ 1026.442272] device veth12f3420 entered promiscuous mode
[ 1026.656393] eth0: renamed from veth579c165
[ 1026.672354] IPv6: ADDRCONF(NETDEV_CHANGE): veth12f3420: link becomes ready
[ 1026.672396] br-888cec1778ee: port 11(veth12f3420) entered blocking state
[ 1026.672397] br-888cec1778ee: port 11(veth12f3420) entered forwarding state
[ 1032.127632] br-888cec1778ee: port 11(veth12f3420) entered disabled state
[ 1032.127798] veth579c165: renamed from eth0
[ 1032.203246] br-888cec1778ee: port 11(veth12f3420) entered disabled state
[ 1032.203813] device veth12f3420 left promiscuous mode
[ 1032.203816] br-888cec1778ee: port 11(veth12f3420) entered disabled state
When I stop the self-hosted sentry instance with docker-compose stop the dmesg messages (and therefore the emails) stop.
install.log docker-compose.log
I think a container is exiting with a non-zero code and therefore has got into a restart loop.
Can you paste output of docker ps -a after running sentry for a while?
Well the geoipupdate container doesn't work as it seems
root•~/self-hosted-22.8.0» docker ps -a [19:39:53]
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0199fad04cfb nginx:1.22.0-alpine "/docker-entrypoint.…" 23 hours ago Up 4 minutes 0.0.0.0:9000->80/tcp, :::9000->80/tcp sentry-self-hosted-nginx-1
c57efa1bf242 getsentry/relay:22.8.0 "/bin/bash /docker-e…" 23 hours ago Up 4 minutes 3000/tcp sentry-self-hosted-relay-1
18ac6fe003c9 sentry-self-hosted-local "/etc/sentry/entrypo…" 23 hours ago Up 4 minutes 9000/tcp sentry-self-hosted-post-process-forwarder-1
e733ae0574e3 sentry-self-hosted-local "/etc/sentry/entrypo…" 23 hours ago Up 4 minutes 9000/tcp sentry-self-hosted-subscription-consumer-transactions-1
aab3d9aa52f8 sentry-self-hosted-local "/etc/sentry/entrypo…" 23 hours ago Up 4 minutes 9000/tcp sentry-self-hosted-cron-1
1ee4895f05c9 sentry-self-hosted-local "/etc/sentry/entrypo…" 23 hours ago Up 4 minutes 9000/tcp sentry-self-hosted-ingest-consumer-1
cf67e1596f5a sentry-self-hosted-local "/etc/sentry/entrypo…" 23 hours ago Up 4 minutes 9000/tcp sentry-self-hosted-worker-1
460ace3d6c90 sentry-cleanup-self-hosted-local "/entrypoint.sh '0 0…" 23 hours ago Up 4 minutes 9000/tcp sentry-self-hosted-sentry-cleanup-1
42e425cd9619 sentry-self-hosted-local "/etc/sentry/entrypo…" 23 hours ago Up 4 minutes 9000/tcp sentry-self-hosted-subscription-consumer-events-1
7fb990969c02 sentry-self-hosted-local "/etc/sentry/entrypo…" 23 hours ago Up 4 minutes (healthy) 9000/tcp sentry-self-hosted-web-1
2f5651e2290f getsentry/snuba:22.8.0 "./docker_entrypoint…" 23 hours ago Up 4 minutes 1218/tcp sentry-self-hosted-snuba-outcomes-consumer-1
d9a4ca3094bf getsentry/snuba:22.8.0 "./docker_entrypoint…" 23 hours ago Up 4 minutes 1218/tcp sentry-self-hosted-snuba-replacer-1
5f56711cc9a5 snuba-cleanup-self-hosted-local "/entrypoint.sh '*/5…" 23 hours ago Up 4 minutes 1218/tcp sentry-self-hosted-snuba-transactions-cleanup-1
f41375f4bc6c snuba-cleanup-self-hosted-local "/entrypoint.sh '*/5…" 23 hours ago Up 4 minutes 1218/tcp sentry-self-hosted-snuba-cleanup-1
0f5143de2038 getsentry/snuba:22.8.0 "./docker_entrypoint…" 23 hours ago Up 1 second 1218/tcp sentry-self-hosted-snuba-transactions-consumer-1
325da397c6af getsentry/snuba:22.8.0 "./docker_entrypoint…" 23 hours ago Up 4 minutes 1218/tcp sentry-self-hosted-snuba-subscription-consumer-events-1
f613e2efbed5 getsentry/snuba:22.8.0 "./docker_entrypoint…" 23 hours ago Up 4 minutes 1218/tcp sentry-self-hosted-snuba-api-1
533aa3ed7bf8 getsentry/snuba:22.8.0 "./docker_entrypoint…" 23 hours ago Up 4 minutes 1218/tcp sentry-self-hosted-snuba-consumer-1
c43efa9a50ce getsentry/snuba:22.8.0 "./docker_entrypoint…" 23 hours ago Up 4 minutes 1218/tcp sentry-self-hosted-snuba-subscription-consumer-transactions-1
8ea6a115ffc1 getsentry/snuba:22.8.0 "./docker_entrypoint…" 23 hours ago Up 4 minutes 1218/tcp sentry-self-hosted-snuba-sessions-consumer-1
fb9541d3ddf7 confluentinc/cp-kafka:5.5.0 "/etc/confluent/dock…" 23 hours ago Up 5 minutes (healthy) 9092/tcp sentry-self-hosted-kafka-1
59f7d154e635 clickhouse-self-hosted-local "/entrypoint.sh" 23 hours ago Up 5 minutes (healthy) 8123/tcp, 9000/tcp, 9009/tcp sentry-self-hosted-clickhouse-1
d03bd0f658d2 confluentinc/cp-zookeeper:5.5.0 "/etc/confluent/dock…" 23 hours ago Up 5 minutes (healthy) 2181/tcp, 2888/tcp, 3888/tcp sentry-self-hosted-zookeeper-1
4faacc71044f symbolicator-cleanup-self-hosted-local "/entrypoint.sh '55 …" 23 hours ago Up 5 minutes 3021/tcp sentry-self-hosted-symbolicator-cleanup-1
779afa6f1256 redis:6.2.4-alpine "docker-entrypoint.s…" 23 hours ago Up 5 minutes (healthy) 6379/tcp sentry-self-hosted-redis-1
ebb87f1f62c7 maxmindinc/geoipupdate:v4.7.1 "/usr/bin/geoipupdat…" 23 hours ago Exited (1) 5 minutes ago sentry-self-hosted-geoipupdate-1
19d65fa2fc74 tianon/exim4 "docker-entrypoint.s…" 23 hours ago Up 5 minutes 25/tcp sentry-self-hosted-smtp-1
8d1c265407a1 memcached:1.6.9-alpine "docker-entrypoint.s…" 23 hours ago Up 5 minutes (healthy) 11211/tcp sentry-self-hosted-memcached-1
1a97588a17ef getsentry/symbolicator:0.5.1 "/bin/bash /docker-e…" 23 hours ago Up 5 minutes 3021/tcp sentry-self-hosted-symbolicator-1
bfb559d8f924 postgres:9.6 "/opt/sentry/postgre…" 24 hours ago Up 5 minutes (healthy) 5432/tcp sentry-self-hosted-postgres-1
Looking into the logs for that specific container I get the following:
root•~/self-hosted-22.8.0» docker logs sentry-self-hosted-geoipupdate-1 [10:34:27]
error loading configuration file /sentry/GeoIP.conf: error opening file: open /sentry/GeoIP.conf: no such file or directory
error loading configuration file /sentry/GeoIP.conf: error opening file: open /sentry/GeoIP.conf: no such file or directory
error loading configuration file /sentry/GeoIP.conf: error opening file: open /sentry/GeoIP.conf: no such file or directory
geoip container does not work out of the box. Further information: https://develop.sentry.dev/self-hosted/geolocation/
But it should not restart itself constantly (and from your log it does not).
So what should I do then? If its not the geoipupdate container then how can we determine which container it is?
Hm, you could install the iproute2 package in the containers, get the veth interface, and compare with the host/logs to find which container it is probably.
So there is no fancy docker command which lists all interfaces and their connected containers?
So I was able to get some more info regarding this problem: The following script can output all interface names of all currently running containers: https://stackoverflow.com/a/60830947
With that I found out, that the container sentry-self-hosted-snuba-transactions-consumer-1 seems to be somehow related to my wg0 interface via a bridge br-804d4ab72990.
My wg0 interface is related to my Wireguard VPN service I have running on my server.
It seems that it was docker related because I now performed the following updates
docker-ce 5:20.10.19~3-0~debian-bullseye
docker-ce-cli 5:20.10.19~3-0~debian-bullseye
docker-ce-rootless-extras 5:20.10.19~3-0~debian-bullseye
docker-compose-plugin 2.11.2~debian-bullseye
and the constant messages are gone. Therefore I will close this issue.