posthog Open-source hobby deploy fails on ubuntu 22

Bug description

I tried the Open-source hobby deploy on a fresh ubuntu 22 hetzner cloud server and ran the /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/posthog/posthog/HEAD/bin/deploy-hobby)" command

the command finished with

⏳ Waiting for PostHog web to boot (this will take a few minutes)
⌛️ PostHog looks up!

🎉🎉🎉  Done! 🎉🎉🎉

To stop the stack run 'docker-compose stop'
To start the stack again run 'docker-compose start'
If you have any issues at all delete everything in this directory and run the curl command again

To upgrade: run /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/posthog/posthog/HEAD/bin/upgrade-hobby)"

PostHog will be up at the location you provided!
https://bellerophon.blackbox.ms

It's dangerous to go alone! Take this: 🦔

unfortunately the application does not start: output of docker ps

CONTAINER ID   IMAGE                                      COMMAND                  CREATED          STATUS                          PORTS                                                                                         NAMES
7bfe8c2ba710   caddy:2.6.1                                "caddy run --config …"   10 minutes ago   Up 10 minutes                   0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 443/udp, 2019/tcp   blackbox-caddy-1
71c9a6a93ca5   posthog/posthog:latest                     "./bin/plugin-server…"   10 minutes ago   Up 9 minutes                    8000-8001/tcp                                                                                 blackbox-plugins-1
435eb6f5a13b   posthog/posthog:latest                     "./bin/docker-worker…"   10 minutes ago   Up 8 minutes                    8000-8001/tcp                                                                                 blackbox-worker-1
0a061801e515   posthog/posthog:latest                     "/compose/start"         10 minutes ago   Up 10 minutes                   8000-8001/tcp                                                                                 blackbox-web-1
8bce46ac9548   posthog/posthog:latest                     "/compose/temporal-d…"   10 minutes ago   Restarting (1) 12 seconds ago                                                                                                 blackbox-temporal-django-worker-1
55c2c13aece6   clickhouse/clickhouse-server:23.4.2.11     "/entrypoint.sh"         10 minutes ago   Up 10 minutes                   8123/tcp, 9000/tcp, 9009/tcp                                                                  blackbox-clickhouse-1
f6274f9af838   temporalio/ui:2.10.3                       "./start-ui-server.sh"   10 minutes ago   Up 10 minutes                   0.0.0.0:8081->8080/tcp, :::8081->8080/tcp                                                     blackbox-temporal-ui-1
701904efdcdb   temporalio/admin-tools:1.20.0              "tail -f /dev/null"      10 minutes ago   Up 10 minutes                                                                                                                 blackbox-temporal-admin-tools-1
a5d712415d89   bitnami/kafka:2.8.1-debian-10-r99          "/opt/bitnami/script…"   10 minutes ago   Up 10 minutes                   9092/tcp                                                                                      blackbox-kafka-1
7f83f0042232   temporalio/auto-setup:1.20.0               "/etc/temporal/entry…"   10 minutes ago   Up 10 minutes                   6933-6935/tcp, 6939/tcp, 7234-7235/tcp, 7239/tcp, 0.0.0.0:7233->7233/tcp, :::7233->7233/tcp   blackbox-temporal-1
1243da6222dc   minio/minio:RELEASE.2022-06-25T15-50-16Z   "sh -c 'mkdir -p /da…"   10 minutes ago   Up 10 minutes                   9000/tcp                                                                                      blackbox-object_storage-1
676f1448f803   zookeeper:3.7.0                            "/docker-entrypoint.…"   10 minutes ago   Up 10 minutes                   2181/tcp, 2888/tcp, 3888/tcp, 8080/tcp                                                        blackbox-zookeeper-1
019d9a22d17a   redis:6.2.7-alpine                         "docker-entrypoint.s…"   10 minutes ago   Up 10 minutes                   6379/tcp                                                                                      blackbox-redis-1
5f53ee0572e4   postgres:12-alpine                         "docker-entrypoint.s…"   10 minutes ago   Up 10 minutes (healthy)         5432/tcp                                                                                      blackbox-db-1
root@bellerophon:/home/blackbox#

docker logs 8bce46ac9548 returns


./bin/temporal-django-worker: line 1: kill: (-7) - No such process
{'event': '\n            You are using the default SECRET_KEY in a production environment!\n            For the safety of your instance, you must generate and set a unique key.,\n            More information on\n            https://posthog.com/docs/self-host/configure/securing-posthog,\n        ', 'timestamp': '2023-06-21T10:02:51.791387Z', 'logger': 'posthog.settings.access', 'level': 'critical', 'pid': 8, 'tid': 139642586904384}
[ERROR] Default SECRET_KEY in production. Stopping Django server…

./bin/temporal-django-worker: line 1: kill: (-7) - No such process
{'event': '\n            You are using the default SECRET_KEY in a production environment!\n            For the safety of your instance, you must generate and set a unique key.,\n            More information on\n            https://posthog.com/docs/self-host/configure/securing-posthog,\n        ', 'timestamp': '2023-06-21T10:03:53.198999Z', 'logger': 'posthog.settings.access', 'level': 'critical', 'pid': 8, 'tid': 140457896224576}
[ERROR] Default SECRET_KEY in production. Stopping Django server…

./bin/temporal-django-worker: line 1: kill: (-7) - No such process
{'event': '\n            You are using the default SECRET_KEY in a production environment!\n            For the safety of your instance, you must generate and set a unique key.,\n            More information on\n            https://posthog.com/docs/self-host/configure/securing-posthog,\n        ', 'timestamp': '2023-06-21T10:04:54.539351Z', 'logger': 'posthog.settings.access', 'level': 'critical', 'pid': 8, 'tid': 140621766002496}
[ERROR] Default SECRET_KEY in production. Stopping Django server…

./bin/temporal-django-worker: line 1: kill: (-7) - No such process
{'event': '\n            You are using the default SECRET_KEY in a production environment!\n            For the safety of your instance, you must generate and set a unique key.,\n            More information on\n            https://posthog.com/docs/self-host/configure/securing-posthog,\n        ', 'timestamp': '2023-06-21T10:05:56.050183Z', 'logger': 'posthog.settings.access', 'level': 'critical', 'pid': 7, 'tid': 140110729320256}
[ERROR] Default SECRET_KEY in production. Stopping Django server…

./bin/temporal-django-worker: line 1: kill: (-6) - No such process
{'event': '\n            You are using the default SECRET_KEY in a production environment!\n            For the safety of your instance, you must generate and set a unique key.,\n            More information on\n            https://posthog.com/docs/self-host/configure/securing-posthog,\n        ', 'timestamp': '2023-06-21T10:06:57.477523Z', 'logger': 'posthog.settings.access', 'level': 'critical', 'pid': 8, 'tid': 140541810083648}
[ERROR] Default SECRET_KEY in production. Stopping Django server…

is there anything i did wrong? i just installed docker and docker-compose and ran the command /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/posthog/posthog/HEAD/bin/deploy-hobby)"

Environment

[ ] PostHog Cloud US, project ID: [please provide from https://app.posthog.com/project/settings#project-id]
[ ] PostHog Cloud EU, project ID: [please provide from https://eu.posthog.com/project/settings#project-id]
[x] PostHog Hobby self-hosted with docker compose, version/commit: latest
[ ] PostHog self-hosted with Kubernetes (deprecated, see "Sunsetting Kubernetes support"), version/commit: [please provide]

Jun 21 '23 10:06 krystofbe

It looks like you don't have the SECRET_KEY environment variable set, it's required for running PostHog, alongside SITE_URL. See docs: https://posthog.com/docs/self-host/configure/environment-variables

Jun 21 '23 16:06 Twixes

Let me know if adding those variables fixes the deployment @krystofbe!

Jun 21 '23 16:06 Twixes

Having similar issues setting up the hobby deploy.

I tried modifying the docker compose manually to set the SECRET_KEY, but then

ubuntu-temporal-django-worker-1  | Traceback (most recent call last):
ubuntu-temporal-django-worker-1  |   File "/python-runtime/django/core/management/__init__.py", line 237, in fetch_command
ubuntu-temporal-django-worker-1  |     app_name = commands[subcommand]
ubuntu-temporal-django-worker-1  | KeyError: 'start_temporal_worker'
ubuntu-temporal-django-worker-1  |
ubuntu-temporal-django-worker-1  | During handling of the above exception, another exception occurred:
ubuntu-temporal-django-worker-1  |
ubuntu-temporal-django-worker-1  | Traceback (most recent call last):
ubuntu-temporal-django-worker-1  |   File "/code/manage.py", line 21, in <module>
ubuntu-temporal-django-worker-1  |     main()
ubuntu-temporal-django-worker-1  |   File "/code/manage.py", line 17, in main
ubuntu-temporal-django-worker-1  |     execute_from_command_line(sys.argv)
ubuntu-temporal-django-worker-1  |   File "/python-runtime/django/core/management/__init__.py", line 419, in execute_from_command_line
ubuntu-temporal-django-worker-1  |     utility.execute()
ubuntu-temporal-django-worker-1  |   File "/python-runtime/django/core/management/__init__.py", line 413, in execute
ubuntu-temporal-django-worker-1  |     self.fetch_command(subcommand).run_from_argv(self.argv)
ubuntu-temporal-django-worker-1  |   File "/python-runtime/django/core/management/__init__.py", line 244, in fetch_command
ubuntu-temporal-django-worker-1  |     settings.INSTALLED_APPS
ubuntu-temporal-django-worker-1  |   File "/python-runtime/django/conf/__init__.py", line 82, in __getattr__
ubuntu-temporal-django-worker-1  |     self._setup(name)
ubuntu-temporal-django-worker-1  |   File "/python-runtime/django/conf/__init__.py", line 69, in _setup
ubuntu-temporal-django-worker-1  |     self._wrapped = Settings(settings_module)
ubuntu-temporal-django-worker-1  |   File "/python-runtime/django/conf/__init__.py", line 170, in __init__
ubuntu-temporal-django-worker-1  |     mod = importlib.import_module(self.SETTINGS_MODULE)
ubuntu-temporal-django-worker-1  |   File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
ubuntu-temporal-django-worker-1  |     return _bootstrap._gcd_import(name[level:], package, level)
ubuntu-temporal-django-worker-1  |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
ubuntu-temporal-django-worker-1  |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
ubuntu-temporal-django-worker-1  |   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
ubuntu-temporal-django-worker-1  |   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
ubuntu-temporal-django-worker-1  |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
ubuntu-temporal-django-worker-1  |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
ubuntu-temporal-django-worker-1  |   File "/code/posthog/settings/__init__.py", line 25, in <module>
ubuntu-temporal-django-worker-1  |     from posthog.settings.celery import *
ubuntu-temporal-django-worker-1  |   File "/code/posthog/settings/celery.py", line 6, in <module>
ubuntu-temporal-django-worker-1  |     from posthog.settings.data_stores import REDIS_URL
ubuntu-temporal-django-worker-1  |   File "/code/posthog/settings/data_stores.py", line 95, in <module>
ubuntu-temporal-django-worker-1  |     raise ImproperlyConfigured(
ubuntu-temporal-django-worker-1  | django.core.exceptions.ImproperlyConfigured: The environment vars "DATABASE_URL" or "POSTHOG_DB_NAME" are absolutely required to run this software
ubuntu-temporal-django-worker-1  | ./bin/temporal-django-worker: line 1: kill: (-6) - No such process

Relevant docker-compose section (only modification was the addition of the secret-key):

    temporal-django-worker:
        command: /compose/temporal-django-worker
        extends:
            file: docker-compose.base.yml
            service: temporal-django-worker
        volumes:
            - ./compose:/compose
        image: posthog/posthog:latest
        environment:
            SENTRY_DSN: 'https://[email protected]/1'
            SITE_URL: https://posthog.ambiencehealthcare.com
            SECRET_KEY: 00000000000000000000000000000000000000000000000000000000

Jun 22 '23 00:06 JDongian

@JDongian Try setting the following postgres and redis environment variables. Started working for me after this

DATABASE_URL: 'postgres://posthog:posthog@db:5432/posthog'
REDIS_URL: 'redis://redis:6379/'

Jul 02 '23 06:07 KarthikReddyPuli

I believe the issue here is in docker-compose.base.yml https://github.com/PostHog/posthog/blob/d53b9d6a62876868099e894eaaa9441d96aa9f27/docker-compose.base.yml#L187-L192

The temporal-django-worker dict extends the &worker reference, but then overwrites the environment to set TEMPORAL_HOST: temporal instead of merging.

Edit: removing that environment variable from docker-compose.base.yml and adding it to docker-compose.yml seems to work for me.

Jul 07 '23 12:07 grncdr

@JDongian Try setting the following postgres and redis environment variables. Started working for me after this
DATABASE_URL: 'postgres://posthog:posthog@db:5432/posthog'
REDIS_URL: 'redis://redis:6379/'

In which file?

Jul 10 '23 12:07 hcbaxiom

I believe the issue here is in docker-compose.base.yml

https://github.com/PostHog/posthog/blob/d53b9d6a62876868099e894eaaa9441d96aa9f27/docker-compose.base.yml#L187-L192

The temporal-django-worker dict extends the &worker reference, but then overwrites the environment to set TEMPORAL_HOST: temporal instead of merging.

Edit: removing that environment variable from docker-compose.base.yml and adding it to docker-compose.yml seems to work for me.

great! can you please elaborate with exact code? would appreciate!!

Jul 10 '23 12:07 hcbaxiom

I believe the issue here is in docker-compose.base.yml

https://github.com/PostHog/posthog/blob/d53b9d6a62876868099e894eaaa9441d96aa9f27/docker-compose.base.yml#L187-L192

The temporal-django-worker dict extends the &worker reference, but then overwrites the environment to set TEMPORAL_HOST: temporal instead of merging.

Edit: removing that environment variable from docker-compose.base.yml and adding it to docker-compose.yml seems to work for me.

===== The above solution worked for me.

Commented these 2 lines in docker-compose.base.yml
```
     #     TEMPORAL_HOST: temporal
```
Added 2 values in docker-compose.yml - TEMPORAL_HOST & SECRET_KEY

My updated docker-compose.base.yml file. My updated docker-compose.yml file

Jul 10 '23 12:07 vikram-ray

hello im still having this issue after this suggested fix... any help?

docker logs fbe838a223d2

./bin/temporal-django-worker: line 1: kill: (-6) - No such process
{'event': '\n            You are using the default SECRET_KEY in a production environment!\n            For the safety of your instance, you must generate and set a unique key.,\n            More information on\n            https://posthog.com/docs/self-host/configure/securing-posthog,\n        ', 'timestamp': '2023-11-06T07:41:54.898522Z', 'logger': 'posthog.settings.access', 'level': 'critical', 'pid': 8, 'tid': 140649049134912}
[ERROR] Default SECRET_KEY in production. Stopping Django server…

./bin/temporal-django-worker: line 1: kill: (-7) - No such process

I believe the issue here is in docker-compose.base.yml https://github.com/PostHog/posthog/blob/d53b9d6a62876868099e894eaaa9441d96aa9f27/docker-compose.base.yml#L187-L192

The temporal-django-worker dict extends the &worker reference, but then overwrites the environment to set TEMPORAL_HOST: temporal instead of merging. Edit: removing that environment variable from docker-compose.base.yml and adding it to docker-compose.yml seems to work for me.

===== The above solution worked for me.
Commented these 2 lines in docker-compose.base.yml
     #     TEMPORAL_HOST: temporal
Added 2 values in docker-compose.yml - TEMPORAL_HOST & SECRET_KEY
My updated docker-compose.base.yml file. My updated docker-compose.yml file

Nov 06 '23 07:11 doudoujay

@doudoujay just faced the same issue, made changes you've referred to, docker-compose up -d and it works

Nov 07 '23 10:11 zytfo

@doudoujay just faced the same issue, made changes you've referred to, docker-compose up -d and it works

@zytfo Hey I run docker-compose restart But it didn't work

Nov 07 '23 18:11 doudoujay

Just adding some additional info: I can confirm that I just ran into the exact same issue today. This was on a fresh selfhost "Hobby" install, using git commit SHA 56c8154f045628a00c02a9155bcae8cafbbb9dfa. The fix in https://github.com/PostHog/posthog/issues/16169#issuecomment-1625351591 worked.

Feb 07 '24 11:02 JKolios

Almost 1 year since the issue was opened, it's still very much happening. Fix mentioned in https://github.com/PostHog/posthog/issues/16169#issuecomment-1628849039 works, just remember to docker-compose up -d after fixing your ymls. Very annoying to face and fix bugs on step #1 of deployment.

May 03 '24 06:05 movy

Has this really been broken for a whole year?

Jun 18 '24 21:06 dollannn

Looks like it's been fixed on one of those last month PRs.

Jun 19 '24 14:06 movy

posthog posthog copied to clipboard

Open-source hobby deploy fails on ubuntu 22

Bug description

Environment

posthog
posthog copied to clipboard