topic: ingest-sessions LAG increase permanently.

[piellick]

Self-Hosted Version

23.11.2

CPU Architecture

x86_64

Docker Version

24.0.4

Docker Compose Version

2.20.2

Steps to Reproduce

Hello everyone, since we update to version 23.11.2 we have this kafka topic that grow. For now we can not make the link with a possible root cause all features seem to work.

check kafka metrics snuba-consumers group topic: ingest-sessions

Expected Result

ingest-session lag sould decrease and event should be consumes.

Actual Result

topic: ingest-sessions continue to increase.

docker compose run --rm kafka kafka-consumer-groups --bootstrap-server kafka:9092 --group snuba-consumers --describe
[+] Creating 1/0
✔ Container sentry-self-hosted-zookeeper-1  Running                                                                                                                                                                                     0.0s 

GROUP           TOPIC                PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                  HOST            CLIENT-ID
snuba-consumers ingest-replay-events 0          249592          249592          0               rdkafka-e52f61a0-d155-410d-88b6-5c05b1bb769a /100.64.2.34    rdkafka
snuba-consumers profiles-call-tree   0          -               0               -               rdkafka-9b050aa5-1782-4665-8308-503dbd574d1b /100.64.2.30    rdkafka
snuba-consumers processed-profiles   0          -               0               -               rdkafka-170f1ccf-7268-4374-a4c2-74afcd2247db /100.64.2.29    rdkafka
snuba-consumers outcomes             0          20706235        20706237        2               rdkafka-aeb36c21-934d-4474-b001-3a3fb11815d0 /100.64.2.17    rdkafka
snuba-consumers events               0          11867802        11867803        1               rdkafka-490d213d-c04b-4bd7-a0a3-f44cc4b49094 /100.64.2.14    rdkafka
snuba-consumers ingest-sessions      0          9922868         10083926        161058          -                                            -               -

Event ID

No response

[piellick]

hello again, could be related to this deprecated feature ? https://github.com/getsentry/self-hosted/issues/2200

thanks a lot.

[hubertdeng123]

We removed all sessions infrastructure from self-hosted. Are you still running any of it? Otherwise, you should be fine to delete that topic.

[piellick]

hey @hubertdeng123 thanks for your help :pray:

How check if sessions are still active ? what is the cleanest way to remove all references to sessions?

Thanks and good holidays :partying_face:

[hubertdeng123]

@piellick Happy holidays to you too!

You can do this by removing the kafka topic ingest-sessions. You should also remove any reference to sessions in your docker compose yaml file. If you're on 23.11.2, sessions should be removed completely already.

[piellick]

thanks @hubertdeng123 :pray: and is it similar for snuba-commit-log ? This one is still increasing .

[azaslavsky]

I think snuba-commit-log should stick around, but @hubertdeng123 do you know more?

[hubertdeng123]

Yep, I think snuba-commit-log should stay.

[piellick]

thanks @azaslavsky @hubertdeng123 .. but isn’t it strange that it’s full?

[azaslavsky]

That is odd - can you look at the logs for that instance, and look for any repeating events? Also, when you restart that container, does this continue to happen?

[piellick]

Hi @azaslavsky , logs on sentry-self-hosted-subscription-consumer-events-1 are rather empty. Nothing since 3days (and more).

We will perform a CT stop/start soon and i'll keep you inform.

[piellick]

Hi @azaslavsky, we did a restart of subscription-consumer-events ct and topic is still full.

[hubertdeng123]

What is your load on your Sentry instance. Unless something is abnormal about your setup, I'm not sure if the topic having a lot of events is a problem.

[piellick]

hi @hubertdeng123 , load is qiute low.

After a second restart, topic is emptied completely :shamrock: and we don't know why. Thanks

[hubertdeng123]

Glad the topic was able to be emptied. Is your problem solved? @piellick

[getsantry[bot]]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀