posthog
posthog copied to clipboard
PostHog
Docker plugin server crashing
Bug Description
Bug description
After using the init script (/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/posthog/posthog/HEAD/bin/deploy-hobby)") and pulling a brand new instance, the plugin keeps crashing:
plugins-1 | 2024-08-16T10:42:59.509498Z [warning ] Skipping async migrations setup. This is unsafe in production! [posthog.apps] pid=23 tid=140091374233408
plugins-1 | 2024-08-16T10:42:59.515049Z [info ] AXES: BEGIN LOG [axes.apps] pid=23 tid=140091374233408
plugins-1 | 2024-08-16T10:42:59.517338Z [info ] AXES: Using django-axes version 5.9.0 [axes.apps] pid=23 tid=140091374233408
plugins-1 | 2024-08-16T10:42:59.517705Z [info ] AXES: blocking by IP only. [axes.apps] pid=23 tid=140091374233408
plugins-1 | Traceback (most recent call last):
plugins-1 | File "/python-runtime/urllib3/connection.py", line 174, in _new_conn
plugins-1 | conn = connection.create_connection(
plugins-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
plugins-1 | File "/python-runtime/urllib3/util/connection.py", line 95, in create_connection
plugins-1 | raise err
plugins-1 | File "/python-runtime/urllib3/util/connection.py", line 85, in create_connection
plugins-1 | sock.connect(sa)
plugins-1 | ConnectionRefusedError: [Errno 111] Connection refused
plugins-1 |
plugins-1 | During handling of the above exception, another exception occurred:
plugins-1 |
plugins-1 | Traceback (most recent call last):
plugins-1 | File "/python-runtime/urllib3/connectionpool.py", line 715, in urlopen
plugins-1 | httplib_response = self._make_request(
plugins-1 | ^^^^^^^^^^^^^^^^^^^
plugins-1 | File "/python-runtime/urllib3/connectionpool.py", line 416, in _make_request
plugins-1 | conn.request(method, url, **httplib_request_kw)
plugins-1 | File "/python-runtime/urllib3/connection.py", line 244, in request
plugins-1 | super(HTTPConnection, self).request(method, url, body=body, headers=headers)
plugins-1 | File "/usr/local/lib/python3.11/http/client.py", line 1303, in request
plugins-1 | self._send_request(method, url, body, headers, encode_chunked)
plugins-1 | File "/usr/local/lib/python3.11/http/client.py", line 1349, in _send_request
plugins-1 | self.endheaders(body, encode_chunked=encode_chunked)
plugins-1 | File "/usr/local/lib/python3.11/http/client.py", line 1298, in endheaders
plugins-1 | self._send_output(message_body, encode_chunked=encode_chunked)
plugins-1 | File "/usr/local/lib/python3.11/http/client.py", line 1058, in _send_output
plugins-1 | self.send(msg)
plugins-1 | File "/usr/local/lib/python3.11/http/client.py", line 996, in send
plugins-1 | self.connect()
plugins-1 | File "/python-runtime/urllib3/connection.py", line 205, in connect
plugins-1 | conn = self._new_conn()
plugins-1 | ^^^^^^^^^^^^^^^^
plugins-1 | File "/python-runtime/urllib3/connection.py", line 186, in _new_conn
plugins-1 | raise NewConnectionError(
plugins-1 | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f694af588d0>: Failed to establish a new connection: [Errno 111] Connection refused
plugins-1 |
plugins-1 | During handling of the above exception, another exception occurred:
plugins-1 |
plugins-1 | Traceback (most recent call last):
plugins-1 | File "/python-runtime/requests/adapters.py", line 564, in send
plugins-1 | resp = conn.urlopen(
plugins-1 | ^^^^^^^^^^^^^
plugins-1 | File "/python-runtime/urllib3/connectionpool.py", line 799, in urlopen
plugins-1 | retries = retries.increment(
plugins-1 | ^^^^^^^^^^^^^^^^^^
plugins-1 | File "/python-runtime/urllib3/util/retry.py", line 592, in increment
plugins-1 | raise MaxRetryError(_pool, url, error or ResponseError(cause))
plugins-1 | urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='clickhouse', port=8123): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f694af588d0>: Failed to establish a new connection: [Errno 111] Connection refused'))
plugins-1 |
plugins-1 | During handling of the above exception, another exception occurred:
plugins-1 |
plugins-1 | Traceback (most recent call last):
plugins-1 | File "/code/manage.py", line 22, in <module>
plugins-1 | main()
plugins-1 | File "/code/manage.py", line 18, in main
plugins-1 | execute_from_command_line(sys.argv)
plugins-1 | File "/python-runtime/django/core/management/__init__.py", line 442, in execute_from_command_line
plugins-1 | utility.execute()
plugins-1 | File "/python-runtime/django/core/management/__init__.py", line 436, in execute
plugins-1 | self.fetch_command(subcommand).run_from_argv(self.argv)
plugins-1 | File "/python-runtime/django/core/management/base.py", line 412, in run_from_argv
plugins-1 | self.execute(*args, **cmd_options)
plugins-1 | File "/python-runtime/django/core/management/base.py", line 458, in execute
plugins-1 | output = self.handle(*args, **options)
plugins-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
plugins-1 | File "/code/posthog/management/commands/migrate_clickhouse.py", line 53, in handle
plugins-1 | self.migrate(CLICKHOUSE_HTTP_URL, options)
plugins-1 | File "/code/posthog/management/commands/migrate_clickhouse.py", line 56, in migrate
plugins-1 | database = Database(
plugins-1 | ^^^^^^^^^
plugins-1 | File "/python-runtime/infi/clickhouse_orm/database.py", line 118, in __init__
plugins-1 | self.db_exists = self._is_existing_database()
plugins-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
plugins-1 | File "/python-runtime/infi/clickhouse_orm/database.py", line 440, in _is_existing_database
plugins-1 | r = self._send("SELECT count() FROM system.databases WHERE name = '%s'" % self.db_name)
plugins-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
plugins-1 | File "/python-runtime/infi/clickhouse_orm/database.py", line 391, in _send
plugins-1 | r = self.request_session.post(self.db_url, params=params, data=data, stream=stream, timeout=self.timeout)
plugins-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
plugins-1 | File "/python-runtime/requests/sessions.py", line 637, in post
plugins-1 | return self.request("POST", url, data=data, json=json, **kwargs)
plugins-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
plugins-1 | File "/python-runtime/requests/sessions.py", line 589, in request
plugins-1 | resp = self.send(prep, **send_kwargs)
plugins-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
plugins-1 | File "/python-runtime/requests/sessions.py", line 703, in send
plugins-1 | r = adapter.send(request, **kwargs)
plugins-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
plugins-1 | File "/python-runtime/requests/adapters.py", line 597, in send
plugins-1 | raise ConnectionError(e, request=request)
plugins-1 | requests.exceptions.ConnectionError: HTTPConnectionPool(host='clickhouse', port=8123): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f694af588d0>: Failed to establish a new connection: [Errno 111] Connection refused'))
also this:
plugins-1 | {"level":"error","time":1723805215435,"pid":49,"hostname":"2866813c627c","errno":-111,"code":"ECONNREFUSED","syscall":"connect","address":"127.0.0.1","port":6479,"msg":"[MAIN] 🔴 Redis error encountered! Trying to reconnect...\n"}
plugins-1 | {"level":"error","time":1723805215525,"pid":49,"hostname":"2866813c627c","errno":-111,"code":"ECONNREFUSED","syscall":"connect","address":"127.0.0.1","port":6479,"msg":"[MAIN] 😡 Redis error encountered! Enough of this, I quit!\n"}
plugins-1 | {"level":"error","time":1723805215525,"pid":49,"hostname":"2866813c627c","msg":"[MAIN] ⏲ Shutting plugin server down gracefully with SIGTERM..."}
How to reproduce
- Run
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/posthog/posthog/HEAD/bin/deploy-hobby)" - Navigate to domain
Additional context
Debug info
- [ ] PostHog Cloud, Debug information: [please copy/paste from https://us.posthog.com/settings/project-details#variables or https://eu.posthog.com/settings/project-details#variables]
- [x] PostHog Hobby self-hosted with `docker compose`, version/commit: latest
- [ ] PostHog self-hosted with Kubernetes (deprecated, see [`Sunsetting Kubernetes support`](https://posthog.com/blog/sunsetting-helm-support-posthog)), version/commit: [please provide]
@FaFre I've encountered this issue as well and you can take a look at these:
- https://github.com/PostHog/posthog/blob/master/plugin-server/src/config/config.ts#L190-L192
- https://github.com/PostHog/posthog/blob/master/docker-compose.dev.yml#L40-L45
- https://github.com/PostHog/posthog/commit/524ebfffb69f6583eca5312e83612a3128681206
It seems like they’ve recently updated the only the dev deployment to the latest, added another redis container
However when using the script, it takes the hoppy deployment which hasn't been updated. So, I switched back to an older version, as this one (d009c8a5b3e1310947711f543ebd2ee63984a1bc) - Its an older version from their Dockerhub before the commit above. Just tested out and its working fine so far if you're just want to test out and wait for them to update.
Or may be try to add the other container (redis7) like in the docker-compose.dev.yml
I've encountered the same issue after a recent upgrade. If somebody does not want to downgrade PostHog back to the version mentioned above, a quick solution is to modify the docker-compose.yml like so:
redis:
extends:
file: docker-compose.base.yml
service: redis
volumes:
- redis-data:/data
+
+ redis7:
+ extends:
+ file: docker-compose.base.yml
+ service: redis7
+ volumes:
+ - redis7-data:/data
...
plugins:
extends:
file: docker-compose.base.yml
service: plugins
image: posthog/posthog:latest
environment:
SENTRY_DSN: https://[email protected]/1
OBJECT_STORAGE_ACCESS_KEY_ID: 'object_storage_root_user'
OBJECT_STORAGE_SECRET_ACCESS_KEY: 'object_storage_root_password'
OBJECT_STORAGE_ENDPOINT: http://objectstorage:19000
OBJECT_STORAGE_ENABLED: true
+ CDP_REDIS_HOST: 'redis7'
+ CDP_REDIS_PORT: 6379
depends_on:
- db
- redis
+ - redis7
- clickhouse
- kafka
- objectstorage
...
volumes:
zookeeper-data:
zookeeper-datalog:
zookeeper-logs:
objectstorage:
postgres-data:
clickhouse-data:
caddy-data:
caddy-config:
redis-data:
+ redis7-data:
kafka-data:
Thanks a lot guys, after some trying I eventually gave up and decided not to use PostHog. It appears too complex and unstable for self-hosting, and the thought of taking it into production was daunting. I hope the team addresses this issue soon, as it has been affecting many users and has been open for almost two weeks. Good luck!
This PR should fix the problem https://github.com/PostHog/posthog/pull/24777
In the meantime, you can fix in your own setup this by adding CDP_REDIS_PORT: 6379 to the plugins service in docker-compose.yml and running
docker compose down plugins
docker compose up plugins -d
