Self-Host deployments not working at all - Problem with /api/billing-v2

[DamonGuzman]

https://github.com/PostHog/posthog/assets/106456287/1f3569c8-f744-44aa-b17b-e358e7133376

Bug Description

#21957 appears to have broken new self-hosted deployments.

When deploying a new self hosted instance, the following error is encounter by the 'web' container.

2024-05-08T19:00:05.491971Z [info     ] request_finished               [django_structlog.middlewares.request] code=200 host=<redacted> ip=<redacted> pid=183 request=<WSGIRequest: GET '/api/projects/1/insights/?order=-last_modified_at&limit=30&offset=0&saved=true&basic=true&include_query_insights=true'> request_id=32c718fb-da3a-4709-9a0e-f3c1c661bd7b team_id=1 tid=138013675977024 user_id=1 x_forwarded_for=<redacted>
2024-05-08T19:00:05.547719Z [info     ] request_started                [django_structlog.middlewares.request] host=<redacted> ip=<redacted> pid=203 request=<WSGIRequest: GET '/api/projects/1/event_definitions?limit=50&event_type=event_custom'> request_id=d08e8284-7792-45d2-8630-b4003cd3c746 team_id=1 tid=138013675977024 user_agent=<redacted> x_forwarded_for=<redacted>
2024-05-08T19:00:05.555322Z [info     ] request_finished               [django_structlog.middlewares.request] code=200 host=<redacted> ip=<redacted> pid=193 request=<WSGIRequest: GET '/api/organizations/@current/plugins/repository/'> request_id=d0b2241f-a015-4fcf-833d-f4bbd58bafb2 team_id=1 tid=138013675977024 user_id=1 x_forwarded_for=<redacted>
2024-05-08T19:00:05.570842Z [info     ] request_finished               [django_structlog.middlewares.request] code=200 host=<redacted> ip=<redacted> pid=203 request=<WSGIRequest: GET '/api/projects/1/event_definitions?limit=50&event_type=event_custom'> request_id=d08e8284-7792-45d2-8630-b4003cd3c746 team_id=1 tid=138013675977024 user_id=1 x_forwarded_for=<redacted> 
2024-05-08T19:00:05.840338Z [error    ] 'tiers'                        [posthog.exceptions] host=<redacted> ip=<redacted> path=/api/billing-v2/ pid=213 request_id=91eeb220-1997-466c-9013-369a53dbd07c team_id=1 tid=138013675977024 x_forwarded_for=<redacted>
Traceback (most recent call last):
  File "/python-runtime/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/code/ee/api/billing.py", line 56, in list
    response = BillingManager(license).get_billing(org, plan_keys)
  File "/code/ee/billing/billing_manager.py", line 170, in get_billing
    if product["tiers"]:
KeyError: 'tiers'
2024-05-08T19:00:06.067494Z [info     ] request_finished               [django_structlog.middlewares.request] code=500 host=<redacted> ip=<redacted> pid=213 request=<WSGIRequest: GET '/api/billing-v2/'> request_id=91eeb220-1997-466c-9013-369a53dbd07c team_id=1 tid=138013675977024 user_id=1 x_forwarded_for=<redacted>
2024-05-08T19:00:06.067863Z [error    ] Internal Server Error: /api/billing-v2/ [django.request] host=<redacted> pid=213 team_id=1 tid=138013675977024 x_forwarded_for=<redacted>
2024-05-08T19:00:06.067951Z [error    ] Internal Server Error: /api/billing-v2/ [django.request] host=<redacted> pid=213 team_id=1 tid=138013675977024 x_forwarded_for=<redacted>

Specifically, the error occurs when calling the /api/billing-v2/

By reverting back to a Docker Image created prior to this PR, the deployment works as expected.

How to reproduce

1. Create self-host deployment
2. Implement workaround discussed in https://github.com/PostHog/posthog/issues/16169#issuecomment-1628849039 for PostHog to run at all.
3. Try to open the PostHog web UI.

Debug info

- [ ] PostHog Cloud, Debug information: [please copy/paste from https://us.posthog.com/settings/project-details#variables]
- [x] PostHog Hobby self-hosted with `docker compose`, version/commit: latest
- [ ] PostHog self-hosted with Kubernetes (deprecated, see [`Sunsetting Kubernetes support`](https://posthog.com/blog/sunsetting-helm-support-posthog)), version/commit: [please provide]

[xrdt]

Thanks for the bug submission, I'll take a look and make a fix.

[ttmx]

Can confirm, am affected by the same issue.

[xrdt]

Hello, this was merged yesterday: https://github.com/PostHog/posthog/pull/22203

@ttmx you commented recently, so I assume that PR didn't actually fix things?

[ttmx]

It did not! I'm not home right now, but I got an Key Error on the line right after one of the fixed lines. if not addon["subscribed"]:. I did have the latest docker image, I quadruple checked.

[DamonGuzman]

@xrdt

I added a comment to #22203 with what I suspect the issue is. I hope this helps 😁

[feedanal]

Another potential culprit: https://github.com/PostHog/posthog/commit/a266ffc60e12f77109558c37abb9cb62a41116d2

Currently we deploy f1d32e6969f531577b32411e985d007f821643f6 (right before https://github.com/PostHog/posthog/commit/a266ffc60e12f77109558c37abb9cb62a41116d2) and it works fine.

[ttmx]

Another potential culprit: https://github.com/PostHog/posthog/commit/a266ffc60e12f77109558c37abb9cb62a41116d2

Currently we deploy f1d32e6969f531577b32411e985d007f821643f6 (right before https://github.com/PostHog/posthog/commit/a266ffc60e12f77109558c37abb9cb62a41116d2) and it works fine.

Thank you for the suggestion, I'll use the referenced release. Would be nice if there were stable releases that weren't just git commits, but I understand it takes work, and I won't demand that from anyone.

[ttmx]

That commit worked for me! Thank you!

[Predhin]

Is there a workaround for this issue ?

[Krishnadas-KP]

Another potential culprit: a266ffc

Currently we deploy f1d32e6969f531577b32411e985d007f821643f6 (right before a266ffc) and it works fine.

This version fixed it for me. Thanks

[xrdt]

Another fix is merged now: https://github.com/PostHog/posthog/pull/22280#issuecomment-2110260502

Mind trying it on for size?

[feedanal]

I tried pulling latest PH container and it works fine! thanks @xrdt for a prompt fix , we were really stuck here

[xrdt]

Thank you for raising, I’ll close the issue now.