Sentry stops logging issues for projects after some time

[leokster]

Self-Hosted Version

24.2.0

CPU Architecture

x86_64

Docker Version

n.a.

Docker Compose Version

n.a.

Steps to Reproduce

Deployed sentry in a K8s cluster. Create a project and start sending events,

import sentry_sdk
from sentry_sdk import capture_exception
import time

sentry_sdk.init(
    dsn="https://[email protected]/3",
    debug=True,
)

while True:
    current_time = time.strftime("%H:%M:%S")
    print(capture_exception(Exception(f"test exception at {current_time}")))
    time.sleep(60)

Expected Result

getting the events logged and shown in the UI forever.

Actual Result

After few minutes (~30) the logging events for this project doesn't work anymore. Events still get sent to the envelope endpoint and I receive a event ID. All services are healthy and no single container crashes. The events just disappear. If I then create a new project and change the DSN everything works again and events get logged to the new project again for some time until it fails again.

Is there anything I can do to investigate e.g. the kafka logs / topics?

Event ID

No response

[leokster]

If I monitor the events kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic ingest-events I only see the event getting through which then also gets displayed in the UI

EDIT

I could find a fix by setting the mode of the relay to proxy but that should probably not be the way to go.

[azaslavsky]

It seems like the events are getting dropped between some of the services - either on the way to consumer, or somewhere else along the route.

Unfortunately, we explicitly only support docker-compose.yml setups. There are other unaffiliated repositories that have k8s setups, but we don't offer support for them.

[oniram88]

I encountered the same thing. it seems that after some time the upstream IP of the relay changes and another service responds (sentry-self-hosted-subscription-consumer-generic-metrics-1 in my case)

checking the various containers they didn't restart, so technically the internal IP address shouldn't change. restarting only the nginx container returns it to working correctly (until the next stop)

[azaslavsky]

Does the nginx fix directly above not work? I'm not sure what would cause the relay IP to change on your machine/docker setup, but this doesn't happen for us on our dogfood instance.

[chipzzz]

Is this related to https://github.com/getsentry/relay/pull/3361 ? ingest-performance-metrics is dropped from relay in chart 23.5.1

[hubertdeng123]

The ingestion of events are separate here @chipzzz. I'm going to close this issue as we don't support k8s deployments of self-hosted officially.

[erlangparasu]

I encountered the same thing. it seems that after some time the upstream IP of the relay changes and another service responds (sentry-self-hosted-subscription-consumer-generic-metrics-1 in my case)

checking the various containers they didn't restart, so technically the internal IP address shouldn't change. restarting only the nginx container returns it to working correctly (until the next stop)

@oniram88 Thanks! Restarting the nginx container works for me. NOTE: i also add cron task to restart only nginx container everyday