Posthog Selfhosted Version Events stopped showing up in events_explorer

[DishantPal]

We are running selfhosted posthog instance on our GCP server.

Now from some time the events have stopped appearing in the events_explorerl tab. ref: https://img.enacton.com/ShareX/2024/02/NkSrt7Wfj9.png

Every time this issue occurs we have to restart the containers and then its started working.

Upon further debugging we have found out that the events are sent from the browser but not received in the server. Ref: https://img.enacton.com/ShareX/2024/02/Qk6QU9jfvi.png

This is the version I can found for the docker running ref: https://img.enacton.com/ShareX/2024/02/t6AjuKKOaz.png ref2: https://img.enacton.com/ShareX/2024/02/kDHkFJwS1A.png

Another thing I've found is that Plugin Server Alive setting is set to "NO" https://img.enacton.com/ShareX/2024/02/cz7XiYREJ7.png May be this would be responsible for the missing events but couldn't figure out why it is configured to "NO"

It would be immensely helpful to know how to debug this issue further.

Thank You.

[mariusandra]

Hi, the plugin server is the one that ingests events after they are received over HTTP. If it's down, no events will come through to the event explorer. However why it is down, we can't say. You're going to have to dig deeper into logs to figure it out.

[DishantPal]

Hi @mariusandra

Appreciate for the detailed response you have shared.

What are some of the steps that we can take to debug and identify the root cause for this? What containers should we inspect for logs? And also what to expect in the logs to identify the cause for this?

Also is there any way that we can add alerting on this specific service to ping when it would be down and restart the container?

Thank You.

[mariusandra]

Check the logs for the plugin server containers. What you'll find there will determine the next steps. I don't know what you might find, all I know is that the plugin server handles ingestion, and if it's down, ingestion is down.

Regarding alerting, kubernetes support is sadly out of scope for us.

[DishantPal]

Hi @mariusandra

Also, have found this documentation to understand how it works, ref: https://posthog.com/docs/how-posthog-works/ingestion-pipeline#squeak-questions

These are the containers that I can see active with the instance, ref: https://img.enacton.com/ShareX/2024/02/zuH1D7EJXD.png Think this is the container for the plugin server: docker logs leafydoc-posthog-plugins-1

Here are the logs, ref: https://img.enacton.com/ShareX/2024/02/putty_5mYEbQ2LOf.png

Currently have restarted the container and it seems to working at this moment.

Do let me know if this is where I can find.

Thank You.

[DishantPal]

Hi, It has occurred again. There are no logs in this container.

ref: https://img.enacton.com/ShareX/2024/02/putty_TgUrhiinNX.png