self-hosted
self-hosted copied to clipboard
getsentry
Failed to create topic events
Self-Hosted Version
22.9.0.dev0
CPU Architecture
aarch64 (Apple Silicon)
Docker Version
20.10.17
Docker Compose Version
2.10.2
Steps to Reproduce
- clone self-hosted repo
-
./install.sh
Expected Result
Installation works without error.
Actual Result
This appears to be the same problem previously reported:
- https://github.com/getsentry/self-hosted/issues/1092
- https://forum.sentry.io/t/failed-to-create-topic-events/15097
As requested on the forum, I have attached install and kafka logs. I don't see anything wrong in the kafka logs, but I have no idea what to look for.
Comparing the topics that the install log claim could not be created, I can see that they all were in fact created in the kafka log, plus 3 more for which there were no error.
Snipped install.log:
...
2022-09-03 01:32:21,518 Attempting to connect to Kafka (attempt 0)...
2022-09-03 01:32:21,702 Connected to Kafka on attempt 0
2022-09-03 01:32:21,707 Creating Kafka topics...
2022-09-03 01:32:22,716 Failed to create topic events
Traceback (most recent call last):
File "/usr/src/snuba/snuba/utils/manage_topics.py", line 31, in create_topics
future.result()
File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 444, in result
return self.__get_result()
File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.8/site-packages/confluent_kafka/admin/__init__.py", line 235, in _make_topics_result
result = f.result()
File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 437, in result
return self.__get_result()
File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
cimpl.KafkaException: KafkaError{code=_TIMED_OUT,val=-185,str="Failed while waiting for response from broker: Local: Timed out"}
2022-09-03 01:32:22,727 Failed to create topic event-replacements
... continues for many more topics ...
There are a few more notable errors reported in the install log:
01:34:58 [WARNING] sentry.utils.geo: Error opening GeoIP database: /geoip/GeoLite2-City.mmdb
01:34:58 [WARNING] sentry.utils.geo: Error opening GeoIP database in Rust: /geoip/GeoLite2-City.mmdb
...
Error setting up IP address geolocation.
Attempting to bring up the stack after installation complains that kafka is unhealthy:
$ docker compose up -d
[+] Running 29/30
...
⠿ Container sentry-self-hosted-kafka-1 Waiting 183.6s
...
container for service "kafka" is unhealthy
docker compose ps then confirms kafka is unhealthy, and several other services have exited, including geoipupdate with a non-zero exit code:
$ docker compose ps master ◼
NAME COMMAND SERVICE STATUS PORTS
sentry-self-hosted-clickhouse-1 "/entrypoint.sh" clickhouse running (healthy) 8123/tcp, 9000/tcp, 9009/tcp
sentry-self-hosted-cron-1 "/etc/sentry/entrypo…" cron created
sentry-self-hosted-geoipupdate-1 "/usr/bin/geoipupdat…" geoipupdate exited (1)
sentry-self-hosted-ingest-consumer-1 "/etc/sentry/entrypo…" ingest-consumer created
sentry-self-hosted-kafka-1 "/etc/confluent/dock…" kafka running (unhealthy) 9092/tcp
sentry-self-hosted-memcached-1 "docker-entrypoint.s…" memcached running (healthy) 11211/tcp
sentry-self-hosted-nginx-1 "/docker-entrypoint.…" nginx created
sentry-self-hosted-post-process-forwarder-1 "/etc/sentry/entrypo…" post-process-forwarder created
sentry-self-hosted-postgres-1 "/opt/sentry/postgre…" postgres running (healthy) 5432/tcp
sentry-self-hosted-redis-1 "docker-entrypoint.s…" redis running (healthy) 6379/tcp
sentry-self-hosted-relay-1 "/bin/bash /docker-e…" relay created
sentry-self-hosted-sentry-cleanup-1 "/entrypoint.sh '0 0…" sentry-cleanup created
sentry-self-hosted-smtp-1 "docker-entrypoint.s…" smtp running 25/tcp
sentry-self-hosted-snuba-api-1 "./docker_entrypoint…" snuba-api exited (0)
sentry-self-hosted-snuba-cleanup-1 "/entrypoint.sh '*/5…" snuba-cleanup created
sentry-self-hosted-snuba-consumer-1 "./docker_entrypoint…" snuba-consumer exited (0)
sentry-self-hosted-snuba-outcomes-consumer-1 "./docker_entrypoint…" snuba-outcomes-consumer exited (0)
sentry-self-hosted-snuba-replacer-1 "./docker_entrypoint…" snuba-replacer exited (0)
sentry-self-hosted-snuba-sessions-consumer-1 "./docker_entrypoint…" snuba-sessions-consumer exited (0)
sentry-self-hosted-snuba-subscription-consumer-events-1 "./docker_entrypoint…" snuba-subscription-consumer-events exited (0)
sentry-self-hosted-snuba-subscription-consumer-transactions-1 "./docker_entrypoint…" snuba-subscription-consumer-transactions exited (0)
sentry-self-hosted-snuba-transactions-cleanup-1 "/entrypoint.sh '*/5…" snuba-transactions-cleanup created
sentry-self-hosted-snuba-transactions-consumer-1 "./docker_entrypoint…" snuba-transactions-consumer exited (0)
sentry-self-hosted-subscription-consumer-events-1 "/etc/sentry/entrypo…" subscription-consumer-events created
sentry-self-hosted-subscription-consumer-transactions-1 "/etc/sentry/entrypo…" subscription-consumer-transactions created
sentry-self-hosted-symbolicator-1 "/bin/bash /docker-e…" symbolicator running 3021/tcp
sentry-self-hosted-symbolicator-cleanup-1 "/entrypoint.sh '55 …" symbolicator-cleanup running 3021/tcp
sentry-self-hosted-web-1 "/etc/sentry/entrypo…" web created
sentry-self-hosted-worker-1 "/etc/sentry/entrypo…" worker created
sentry-self-hosted-zookeeper-1 "/etc/confluent/dock…" zookeeper running (healthy) 2181/tcp, 2888/tcp, 3888/tcp
After docker compose down and docker compose up -d again, the web service is now unhealthy:
sentry-self-hosted-web-1 | Updating certificates in /etc/ssl/certs...
sentry-self-hosted-web-1 | 0 added, 0 removed; done.
sentry-self-hosted-web-1 | Running hooks in /etc/ca-certificates/update.d...
sentry-self-hosted-web-1 | done.
sentry-self-hosted-web-1 | 02:38:09 [INFO] sentry.plugins.github: apps-not-configured
sentry-self-hosted-web-1 | *** Starting uWSGI 2.0.20 (64bit) on [Sat Sep 3 02:38:12 2022] ***
sentry-self-hosted-web-1 | compiled with version: 8.3.1 20190311 (Red Hat 8.3.1-3) on 11 November 2021 18:13:24
sentry-self-hosted-web-1 | os: Linux-5.10.124-linuxkit #1 SMP PREEMPT Thu Jun 30 08:18:26 UTC 2022
sentry-self-hosted-web-1 | nodename: 6d1430a4eb15
sentry-self-hosted-web-1 | machine: x86_64
sentry-self-hosted-web-1 | clock source: unix
sentry-self-hosted-web-1 | pcre jit disabled
sentry-self-hosted-web-1 | detected number of CPU cores: 8
sentry-self-hosted-web-1 | current working directory: /
sentry-self-hosted-web-1 | detected binary path: /usr/local/bin/python
sentry-self-hosted-web-1 | your memory page size is 4096 bytes
sentry-self-hosted-web-1 | detected max file descriptor number: 1048576
sentry-self-hosted-web-1 | lock engine: pthread robust mutexes
sentry-self-hosted-web-1 | !!! it looks like your kernel does not support pthread robust mutexes !!!
sentry-self-hosted-web-1 | !!! falling back to standard pthread mutexes !!!
sentry-self-hosted-web-1 | thunder lock: enabled
sentry-self-hosted-web-1 | uWSGI http bound on 0.0.0.0:9000 fd 4
sentry-self-hosted-web-1 | uwsgi socket 0 bound to TCP address 127.0.0.1:38655 (port auto-assigned) fd 3
sentry-self-hosted-web-1 | Python version: 3.8.13 (default, Aug 23 2022, 09:42:19) [GCC 10.2.1 20210110]
sentry-self-hosted-web-1 | --- Python VM already initialized ---
sentry-self-hosted-web-1 | Python main interpreter initialized at 0x4000008870
sentry-self-hosted-web-1 | python threads support enabled
sentry-self-hosted-web-1 | your server socket listen backlog is limited to 100 connections
sentry-self-hosted-web-1 | your mercy for graceful operations on workers is 60 seconds
sentry-self-hosted-web-1 | setting request body buffering size to 65536 bytes
sentry-self-hosted-web-1 | mapped 1924224 bytes (1879 KB) for 12 cores
sentry-self-hosted-web-1 | *** Operational MODE: preforking+threaded ***
sentry-self-hosted-web-1 | spawned uWSGI master process (pid: 1916)
sentry-self-hosted-web-1 | spawned uWSGI worker 1 (pid: 1937, cores: 4)
sentry-self-hosted-web-1 | spawned uWSGI worker 2 (pid: 1939, cores: 4)
sentry-self-hosted-web-1 | spawned uWSGI worker 3 (pid: 1940, cores: 4)
sentry-self-hosted-web-1 | spawned uWSGI http 1 (pid: 1943)
sentry-self-hosted-web-1 | Sat Sep 3 02:38:13 2022 - *** uWSGI listen queue of socket "127.0.0.1:38655" (fd: 3) full !!! (33291120/64) ***
sentry-self-hosted-web-1 | Sat Sep 3 02:38:14 2022 - *** uWSGI listen queue of socket "127.0.0.1:38655" (fd: 3) full !!! (33291120/64) ***
sentry-self-hosted-web-1 | Sat Sep 3 02:38:15 2022 - *** uWSGI listen queue of socket "127.0.0.1:38655" (fd: 3) full !!! (33291120/64) ***
sentry-self-hosted-web-1 | Sat Sep 3 02:38:16 2022 - *** uWSGI listen queue of socket "127.0.0.1:38655" (fd: 3) full !!! (33291120/64) ***
sentry-self-hosted-web-1 | Sat Sep 3 02:38:17 2022 - *** uWSGI listen queue of socket "127.0.0.1:38655" (fd: 3) full !!! (33291120/64) ***
... continues ...
web eventually claims to be healthy, while the log is still repeating the same (fd: 3) full !!! every second.
nginx and relay claim to be created. After manually stopping and starting them again, I can finally connect to localhost:9000:
docker compose stop nginx relay
docker compose up -d relay
docker compose up -d nginx
I have also seen these logs appear on my self-hosted setup on apple M1 mac. We're still working to fully support the aarch64 cpu architecture and the install for that may be pretty unreliable.
1. 2022-09-03 01:32:21,518 Attempting to connect to Kafka (attempt 0)...
2022-09-03 01:32:21,702 Connected to Kafka on attempt 0
2022-09-03 01:32:21,707 Creating Kafka topics...
2022-09-03 01:32:22,716 Failed to create topic events
...
2. container for service "kafka" is unhealthy
3. 01:34:58 [WARNING] sentry.utils.geo: Error opening GeoIP database: /geoip/GeoLite2-City.mmdb
01:34:58 [WARNING] sentry.utils.geo: Error opening GeoIP database in Rust: /geoip/GeoLite2-City.mmdb
...
Error setting up IP address geolocation.
It seems like my local setup is working fine even though some of these errors pop up now and then. @ethanhs have you seen these before?
Yeah the GeoIP database errors are because we don't add the databases by default, you have to add them yourself https://develop.sentry.dev/self-hosted/geolocation/
It looks like in the install log kafka does eventually come up and set up topics. If this is a fresh install can you try running reset.sh? This is destructive, so only do this if sentry isn't storing any important data.
This issue has gone three weeks without activity. In another week, I will close it.
But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!
"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}