self-hosted icon indicating copy to clipboard operation
self-hosted copied to clipboard

Published 20 hours ago • verified publisher icon getsentry

Kafka error when sending a large number of transactions

Open GoldUser32 opened this issue 8 months ago • 20 comments

Self-Hosted Version

Self-Hosted Version Sentry 25.2.0

CPU Architecture

x86_64

Docker Version

28.0.1

Docker Compose Version

2.33.1

Machine Specification

  • [x] My system meets the minimum system requirements of Sentry

Steps to Reproduce

When the applications sends about 600k transactions in hour to sentry. I get a transaction-consumer container error

I've I tried cleaning out the kafka kafka-consumer-groups --bootstrap-server kafka:9092 --all-groups --all-topics --reset-offsets --to-latest --execute also tried stopping the containers, deleting the kafka volume, and restarting again .install.sh. This works for a while. then the transactions don't appear in sentry again.

my sentry .env file include SENTRY_EVENT_RETENTION_DAYS=30

from docker-compose.yml kafka: <<: *restart_policy image: "confluentinc/cp-kafka:7.6.1" # ports: # - 9092 environment: # https://docs.confluent.io/platform/current/installation/docker/config-reference.html#cp-kakfa-example KAFKA_PROCESS_ROLES: "broker,controller" KAFKA_CONTROLLER_QUORUM_VOTERS: "[email protected]:29093" KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER" KAFKA_NODE_ID: "1001" CLUSTER_ID: "MkU3OEVBNTcwNTJENDM2Qk" KAFKA_LISTENERS: "PLAINTEXT://0.0.0.0:29092,INTERNAL://0.0.0.0:9093,EXTERNAL://0.0.0.0:9092,CONTROLLER://0.0.0.0:29093" KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://127.0.0.1:29092,INTERNAL://kafka:9093,EXTERNAL://kafka:9092" KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "PLAINTEXT:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT" KAFKA_INTER_BROKER_LISTENER_NAME: "PLAINTEXT" KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: "1" KAFKA_OFFSETS_TOPIC_NUM_PARTITIONS: "1" #KAFKA_LOG_CLEANUP_POLICY: delete KAFKA_LOG_CLEANER_ENABLE: true KAFKA_LOG_CLEANUP_POLICY: delete KAFKA_LOG_RETENTION_HOURS: "12" KAFKA_MESSAGE_MAX_BYTES: "700000000" #50MB or bust KAFKA_MAX_REQUEST_SIZE: "600000000" #50MB on requests apparently too # KAFKA_MAX_RECORDS_PER_USER_OP: CONFLUENT_SUPPORT_METRICS_ENABLE: "false" KAFKA_LOG4J_LOGGERS: "kafka.cluster=WARN,kafka.controller=WARN,kafka.coordinator=WARN,kafka.log=WARN,kafka.server=WARN,state.change.logger=WARN" KAFKA_LOG4J_ROOT_LOGLEVEL: "DEBUG" KAFKA_TOOLS_LOG4J_LOGLEVEL: "DEBUG" ulimits: nofile: soft: 8192 hard: 8192 volumes: - "sentry-kafka:/var/lib/kafka/data" - "sentry-kafka-log:/var/lib/kafka/log" - "sentry-secrets:/etc/kafka/secrets" healthcheck: <<: *healthcheck_defaults test: ["CMD-SHELL", "nc -z localhost 9092"] interval: 10s timeout: 10s retries: 30

from relay/config.yml limits: max_concurrent_requests: 100000 max_concurrent_queries: 1000 max_thread_count: 800

Expected Result

working sentry

Actual Result

“File “/.venv/lib/python3.13/site-packages/arroyo/backends/kafka/consumer.py”, line 422, in poll transactions-consumer-1 | raise OffsetOutOfRange(str(error)) transactions-consumer-1 | arroyo.errors.OffsetOutOfRange: KafkaError{code=_AUTO_OFFSET_RESET,val=-140,str=“fetch failed due to requested offset not available on the broker: Broker: Offset out of range (broker 1001)”}"”

Event ID

No response

GoldUser32 avatar Mar 27 '25 08:03 GoldUser32

Image Image

GoldUser32 avatar Mar 27 '25 08:03 GoldUser32

I also tried doing as described in this issue https://github.com/getsentry/self-hosted/issues/1894#issuecomment-1426538497 it only works for a short time

Image

GoldUser32 avatar Mar 27 '25 09:03 GoldUser32

What is your server specifications? Usually this means the consumers can't keep up with the throughput.

aldy505 avatar Mar 29 '25 06:03 aldy505

Self-Hosted Version Sentry run on baremetal server Delll PowerEdge R630: 2xCPU Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz, 4xDDR4 32Gb, 2xSSD 4T GB. What throughput do you mean?

GoldUser32 avatar Apr 02 '25 07:04 GoldUser32

I update sentry to latest version 25.3.0 and now see errors in clickhouse container

clickhouse-1 | 2025.04.02 09:53:59.938628 [ 535 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 107, Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below): clickhouse-1 | clickhouse-1 | 0. Poco::Net::SocketImpl::error(int, String const&) @ 0x0000000015b3dbf2 in /usr/bin/clickhouse clickhouse-1 | 1. Poco::Net::SocketImpl::peerAddress() @ 0x0000000015b40376 in /usr/bin/clickhouse clickhouse-1 | 2. DB::ReadBufferFromPocoSocket::ReadBufferFromPocoSocket(Poco::Net::Socket&, unsigned long) @ 0x000000000c896cc6 in /usr/bin/clickhouse clickhouse-1 | 3. DB::HTTPServerRequest::HTTPServerRequest(std::shared_ptrDB::IHTTPContext, DB::HTTPServerResponse&, Poco::Net::HTTPServerSession&) @ 0x000000001315451b in /usr/bin/clickhouse clickhouse-1 | 4. DB::HTTPServerConnection::run() @ 0x0000000013152ba4 in /usr/bin/clickhouse clickhouse-1 | 5. Poco::Net::TCPServerConnection::start() @ 0x0000000015b42834 in /usr/bin/clickhouse clickhouse-1 | 6. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b43a31 in /usr/bin/clickhouse clickhouse-1 | 7. Poco::PooledThread::run() @ 0x0000000015c7a667 in /usr/bin/clickhouse clickhouse-1 | 8. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c7893c in /usr/bin/clickhouse clickhouse-1 | 9. ? @ 0x00007f9e36cac609 in ? clickhouse-1 | 10. ? @ 0x00007f9e36bd1353 in ? clickhouse-1 | (version 23.8.11.29.altinitystable (altinity build))

GoldUser32 avatar Apr 02 '25 09:04 GoldUser32

@aldy505 any updates? I tried fix it with changed rust-consumer to consumer in my docker-compose.yml and reduild containers. Also add in clickhouse/config.xml <listen_host>0.0.0.0</listen_host> <listen_host>9000</listen_host> <tcp_port>9000</tcp_port>

GoldUser32 avatar Apr 04 '25 13:04 GoldUser32

Hello. Sorry this fell out of my radar. For that ClickHouse errors, can you ingest events/data just fine? Staying in rust-consumer is fine as long as you can ingest data.

It's a known issue anyway, you can safely ignore it. See https://github.com/getsentry/snuba/issues/5707

aldy505 avatar Apr 04 '25 13:04 aldy505

I see errors only, only in discover. Developers send just this data. After added listen host, ports in config.xml. I have new error in click:<Warning> Application: Listen [9000]:9004 failed: Poco::Exception. Code: 1000, e.code() = 99, Net Exception: Cannot assign requested address. How configure click right? Thank you, I return rust-consumer in docker compose config back now.

Image

GoldUser32 avatar Apr 04 '25 14:04 GoldUser32

After added listen host, ports in config.xml. I have new error in click:<Warning> Application: Listen [9000]:9004 failed: Poco::Exception. Code: 1000, e.code() = 99, Net Exception: Cannot assign requested address.

This is an error on your ClickHouse config. You should remove these (if you still have it):

<listen_host>0.0.0.0</listen_host>
<listen_host>9000</listen_host>
<tcp_port>9000</tcp_port>

Listen Host of 9000 is wrong, since it should be port, not host. By default, ClickHouse listen to all interface (meaning 0.0.0.0 and [::] [the IPv6 equivalent]) on port 9000 for TCP and 8123 for HTTP.

aldy505 avatar Apr 07 '25 09:04 aldy505

I remove it from clickhouse config. right now we're getting 1m transactions per hour. I have tried stopping all containers. start the kafka container separately. clean all tops kafka-consumer-groups --bootstrap-server kafka:9092 --all-groups --all-topics --reset-offsets --to-latest --execute and then start the whole stack of containers. after that the transactions appear in Discover for a few minutes and then disappear again. can you suggest something?

GoldUser32 avatar Apr 23 '25 09:04 GoldUser32

@aldy505 ?

GoldUser32 avatar Apr 29 '25 12:04 GoldUser32

I have tried stopping all containers. start the kafka container separately. clean all tops kafka-consumer-groups --bootstrap-server kafka:9092 --all-groups --all-topics --reset-offsets --to-latest --execute and then start the whole stack of containers. after that the transactions appear in Discover for a few minutes and then disappear again.

I would like to see some Docker container logs if this happened again. Can you provide me with some?

aldy505 avatar May 02 '25 06:05 aldy505 

kafka-1  | [2025-05-02 09:40:31,450] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267680. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267681. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-38f8b7f9-7e5b-49cd-83a3-83b813781627, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267681. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267681. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267682. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267682. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-84fb0e2a-1dd4-4a09-9ac1-060ced2726b8, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267683. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267683. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267683. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267684. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267684. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267685. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-c162d23a-9224-4bb8-aa47-0af99a93fe36, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267685. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267685. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267686. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267686. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267687. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-3a16924a-c302-4559-b454-202b647a3038, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267687. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267687. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267688. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267688. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267689. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-cf488cfa-8482-4cb0-a76f-11af7bd0932a, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267689. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267689. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267690. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267690. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-dc8a9548-d848-4777-b856-fa8118f7b698, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267691. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267691. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267691. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267692. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267692. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267693. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-3d22c105-02ac-4c1e-80b0-a470aba0ac61, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267693. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,451] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267693. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267694. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267694. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267695. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267695. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-3e13bfbc-1281-430e-8749-253d3651003c, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267695. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267696. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267696. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267697. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-562d6ef6-e2b7-443f-805e-a49bd9290f36, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267697. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267697. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267698. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267698. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267699. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-875f5036-2834-49bd-a97a-c8757baa5b0f, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267699. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267699. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267700. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267700. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-3253ddc5-896d-4b4a-af9a-b6ad6d8c72c7, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267701. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267701. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267701. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267702. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267702. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267703. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-27e2cb77-70a1-4e88-8fce-99944adebb91, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267703. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267703. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267704. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267704. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-8dd894a6-4e04-453d-8b96-9eb859d3c8b2, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267705. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267705. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267705. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267706. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267706. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267707. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267707. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-55bd0371-9923-42c7-a34a-48968864eab8, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267707. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,452] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267708. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267708. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-7d856d18-5ac5-42e2-8e6a-f5f369c4c2df, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267709. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267709. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267709. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267710. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267710. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267711. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267711. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-6ec54416-93f1-4e1a-96c2-14fbe02b8e13, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267711. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267712. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267712. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267713. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-a8012408-9a0a-4d38-a079-d90609b5efd8, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267713. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267713. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267714. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267714. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267715. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-171eea31-0742-4e24-a634-6ebaecf89b51, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267715. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267715. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267716. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267716. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267717. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-904412de-4e26-4f8b-a805-24b289558cd5, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267717. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267717. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267718. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267718. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267719. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-ca69d1c8-46c8-49b4-9106-8ff75b2db2a7, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267719. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267719. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267720. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267720. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267721. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267721. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-198aba16-a85f-4d4d-972b-94b3bc4c8395, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267721. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267722. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,453] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267722. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267723. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-c4ede784-b07c-4ae5-861d-acb6d3adb98d, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267723. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267723. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267724. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267724. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267725. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-38617bc7-2d8a-47c1-b9eb-60c3643d9805, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267725. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267725. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267726. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267726. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267727. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-26f25349-03ad-4a10-bfc6-002efdb019c6, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267727. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267727. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267728. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267728. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267729. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267729. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-a6e9c286-052d-426d-a2bd-ab92c88bcd35, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267729. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267730. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267730. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267731. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-24aeed5f-d4fc-409d-804c-93954171c23f, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267731. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267731. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267732. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267732. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267733. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-364eb1fa-990a-4643-87df-23f06e85e4a6, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267733. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267733. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267734. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267734. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267735. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,454] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267735. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,455] INFO Loaded member MemberMetadata(memberId=rdkafka-58124a9a-2efb-4ec9-affb-4aea3fc8f296, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267735. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,455] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267736. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,455] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267736. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,455] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267737. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,455] INFO Loaded member MemberMetadata(memberId=rdkafka-b1ef7ab1-c834-4ae5-a998-7cdfd3942ea4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267737. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,455] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267737. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,455] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267738. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,455] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267738. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,455] INFO Loaded member MemberMetadata(memberId=rdkafka-f93c4abb-058d-4252-be0c-f201a334e743, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.47, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267739. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,455] INFO Loaded member MemberMetadata(memberId=rdkafka-805b5c84-81a6-4e2f-8abd-4005e49896d4, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.45, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267739. (kafka.coordinator.group.GroupMetadata$)
kafka-1  | [2025-05-02 09:40:31,455] INFO Loaded member MemberMetadata(memberId=rdkafka-a108ac83-bbd6-4600-a8ce-ecd9be2aa13f, groupInstanceId=None, clientId=rdkafka, clientHost=/172.18.0.35, sessionTimeoutMs=30000, rebalanceTimeoutMs=30000, supportedProtocols=List(range)) in group ingest-consumer with generation 267739. (kafka.coordinator.group.GroupMetadata$)

GoldUser32 avatar May 02 '25 10:05 GoldUser32

kafka container logs after clearing the topics. what other containers do you need logs for?

GoldUser32 avatar May 02 '25 10:05 GoldUser32

some debug log from kafka DEBUG [BrokerToControllerChannelManager id=1001 name=heartbeat] Sending BROKER_HEARTBEAT request with header RequestHeader(apiKey=BROKER_HEARTBEAT, apiVersion=0, clientId=1001, correlationId=67, headerVersion=2) and timeout 4500 to node 1001: BrokerHeartbeatRequestData(brokerId=1001, brokerEpoch=5000018, currentMetadataOffset=5000104, wantFence=false, wantShutDown=false) (org.apache.kafka.clients.NetworkClient)

DEBUG [QuorumController id=1001] Completing read-only operation processBrokerHeartbeat(1566226049) immediately because the purgatory is empty. (org.apache.kafka.controller.QuorumController)

DEBUG [QuorumController id=1001] Processed processBrokerHeartbeat(1566226049) in 269 us (org.apache.kafka.controller.QuorumController)

DEBUG [QuorumController id=1001] Completing read-only operation processBrokerHeartbeat(1566226049) immediately because the purgatory is empty. (org.apache.kafka.controller.QuorumController)

DEBUG [QuorumController id=1001] Processed processBrokerHeartbeat(1566226049) in 269 us (org.apache.kafka.controller.QuorumController)

DEBUG [BrokerToControllerChannelManager id=1001 name=heartbeat] Received BROKER_HEARTBEAT response from node 1001 for request with header RequestHeader(apiKey=BROKER_HEARTBEAT, apiVersion=0, clientId=1001, correlationId=67, headerVersion=2): BrokerHeartbeatResponseData(throttleTimeMs=0, errorCode=0, isCaughtUp=true, isFenced=false, shouldShutDown=false) (org.apache.kafka.clients.NetworkClient)

GoldUser32 avatar May 02 '25 10:05 GoldUser32

At least these:

  • snuba-transactions-consumer:
  • snuba-subscription-consumer-transactions
  • transactions-consumer
  • post-process-forwarder-transactions
  • subscription-consumer-transactions

aldy505 avatar May 02 '25 10:05 aldy505

snuba-transactions-consumer

snuba-transactions-consumer-1  | 2025-05-02 09:43:49,382 Initializing Snuba...
snuba-transactions-consumer-1  | 2025-05-02 09:44:01,008 Snuba initialization took 11.627038423903286s
snuba-transactions-consumer-1  | 2025-05-02 09:44:02,638 Initializing Snuba...
snuba-transactions-consumer-1  | 2025-05-02 09:44:16,303 Snuba initialization took 13.667024563997984s
snuba-transactions-consumer-1  | {"timestamp":"2025-05-02T09:44:16.323419Z","level":"INFO","fields":{"message":"Storage: transactions, ClickHouse Table Name: transactions_local, Message Processor: MessageProcessorConfig { python_class_name: \"TransactionsMessageProcessor\", python_module: \"snuba.datasets.processors.transactions_processor\" }, ClickHouse host: clickhouse, ClickHouse port: 9000, ClickHouse HTTP port: 8123, ClickHouse database: default"},"target":"rust_snuba::consumer"}
snuba-transactions-consumer-1  | {"timestamp":"2025-05-02T09:44:16.323621Z","level":"INFO","fields":{"message":"Starting consumer for \"transactions\"","storage":"transactions"},"target":"rust_snuba::consumer"}
snuba-transactions-consumer-1  | {"timestamp":"2025-05-02T09:44:19.356069Z","level":"INFO","fields":{"message":"Kafka consumer member id: \"rdkafka-a97e04e4-378a-447c-a074-9314dd3c9445\""},"target":"sentry_arroyo::backends::kafka"}
snuba-transactions-consumer-1  | {"timestamp":"2025-05-02T09:44:19.356317Z","level":"INFO","fields":{"message":"New partitions assigned: {Partition { topic: Topic(\"transactions\"), index: 0 }: 6904206}"},"target":"sentry_arroyo::processing"}

snuba-subscription-consumer-transactions

snuba-subscription-consumer-transactions-1  | 2025-05-02 09:43:50,508 Initializing Snuba...
snuba-subscription-consumer-transactions-1  | 2025-05-02 09:44:03,571 Snuba initialization took 13.064565299078822s
snuba-subscription-consumer-transactions-1  | 2025-05-02 09:44:05,122 Initializing Snuba...
snuba-subscription-consumer-transactions-1  | 2025-05-02 09:44:17,901 Snuba initialization took 12.781019834801555s
snuba-subscription-consumer-transactions-1  | {"module": "builtins", "event": "Checking Clickhouse connections", "severity": "info", "timestamp": "2025-05-02T09:44:17.935787Z"}
snuba-subscription-consumer-transactions-1  | 2025-05-02 09:44:20,970 New partitions assigned: {Partition(topic=Topic(name='snuba-transactions-commit-log'), index=0): 213149}
snuba-subscription-consumer-transactions-1  | 2025-05-02 09:44:20,971 Member id: 'rdkafka-b9e25c3b-aaff-4867-b856-841c122e61a5'

transactions-consumer

transactions-consumer-1  | Updating certificates in /etc/ssl/certs...
transactions-consumer-1  | 0 added, 0 removed; done.
transactions-consumer-1  | Running hooks in /etc/ca-certificates/update.d...
transactions-consumer-1  | done.
transactions-consumer-1  | 09:44:10 [INFO] arroyo.processing.processor: New partitions assigned: {Partition(topic=Topic(name='ingest-transactions'), index=0): 428323463}
transactions-consumer-1  | 09:44:10 [INFO] arroyo.processing.processor: Member id: 'rdkafka-9eb68e89-15a0-4869-bf91-73f014c5feb8'

post-process-forwarder-transactions

post-process-forwarder-transactions-1  | Updating certificates in /etc/ssl/certs...
post-process-forwarder-transactions-1  | 0 added, 0 removed; done.
post-process-forwarder-transactions-1  | Running hooks in /etc/ca-certificates/update.d...
post-process-forwarder-transactions-1  | done.
post-process-forwarder-transactions-1  | 09:44:15 [INFO] arroyo.processing.processor: New partitions assigned: {Partition(topic=Topic(name='transactions'), index=0): 6904206}
post-process-forwarder-transactions-1  | 09:44:15 [INFO] arroyo.processing.processor: Member id: 'rdkafka-562ea2f9-3ade-4d0d-9dbe-b9324484c7f9'
post-process-forwarder-transactions-1  | 09:44:15 [INFO] sentry.post_process_forwarder.post_process_forwarder: Starting multithreaded post process forwarder

subscription-consumer-transactions

subscription-consumer-transactions-1  | Updating certificates in /etc/ssl/certs...
subscription-consumer-transactions-1  | 0 added, 0 removed; done.
subscription-consumer-transactions-1  | Running hooks in /etc/ca-certificates/update.d...
subscription-consumer-transactions-1  | done.
subscription-consumer-transactions-1  | 09:44:13 [INFO] arroyo.processing.processor: New partitions assigned: {Partition(topic=Topic(name='transactions-subscription-results'), index=0): 0}
subscription-consumer-transactions-1  | 09:44:13 [INFO] arroyo.processing.processor: Member id: 'rdkafka-e446a754-0dd9-40be-a826-3d7d6d38a92b'

GoldUser32 avatar May 02 '25 11:05 GoldUser32

I see that I have a lot of lag in the topic ingest-transactions

GROUP           TOPIC               CURRENT-OFFSET  LOG-END-OFFSET  LAG
ingest-consumer ingest-transactions 442771558       443389135       617577

and the container transactions-consumer keeps restarting

GoldUser32 avatar May 03 '25 11:05 GoldUser32

logs transactions-consumer

Running hooks in /etc/ca-certificates/update.d...
transactions-consumer-1  | done.
transactions-consumer-1  | 11:33:07 [INFO] arroyo.processing.processor: New partitions assigned: {Partition(topic=Topic(name='ingest-transactions'), index=0): 442771558}
transactions-consumer-1  | 11:33:07 [INFO] arroyo.processing.processor: Member id: 'rdkafka-00401f69-5657-4163-b22b-b60d5e45c6d7'
transactions-consumer-1  | Traceback (most recent call last):
transactions-consumer-1  |   File "/.venv/lib/python3.13/site-packages/arroyo/processing/processor.py", line 333, in run
transactions-consumer-1  |     self._run_once()
transactions-consumer-1  |     ~~~~~~~~~~~~~~^^
transactions-consumer-1  |   File "/.venv/lib/python3.13/site-packages/arroyo/processing/processor.py", line 396, in _run_once
transactions-consumer-1  |     self.__message = self.__consumer.poll(timeout=1.0)
transactions-consumer-1  |                      ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
transactions-consumer-1  |   File "/.venv/lib/python3.13/site-packages/arroyo/backends/kafka/consumer.py", line 422, in poll
transactions-consumer-1  |     raise OffsetOutOfRange(str(error))
transactions-consumer-1  | arroyo.errors.OffsetOutOfRange: KafkaError{code=_AUTO_OFFSET_RESET,val=-140,str="fetch failed due to requested offset not available on the broker: Broker: Offset out of range (broker 1001)"}
transactions-consumer-1  | 11:33:07 [ERROR] arroyo.processing.processor: Caught exception, shutting down...
transactions-consumer-1  | 11:33:07 [INFO] arroyo.processing.processor: Closing <arroyo.backends.kafka.consumer.KafkaConsumer object at 0x7f196ad4b0e0>...
transactions-consumer-1  | 11:33:07 [INFO] arroyo.processing.processor: Partitions to revoke: [Partition(topic=Topic(name='ingest-transactions'), index=0)]
transactions-consumer-1  | 11:33:07 [INFO] arroyo.processing.processor: Partition revocation complete.
transactions-consumer-1  | 11:33:07 [INFO] arroyo.processing.processor: Processor terminated
transactions-consumer-1  | Traceback (most recent call last):
transactions-consumer-1  |   File "/.venv/bin/sentry", line 4, in <module>
transactions-consumer-1  |     raise SystemExit(main())
transactions-consumer-1  |                      ~~~~^^
transactions-consumer-1  |   File "/usr/src/sentry/src/sentry/runner/main.py", line 144, in main
transactions-consumer-1  |     func(**kwargs)
transactions-consumer-1  |     ~~~~^^^^^^^^^^
transactions-consumer-1  |   File "/.venv/lib/python3.13/site-packages/click/core.py", line 1157, in __call__
transactions-consumer-1  |     return self.main(*args, **kwargs)
transactions-consumer-1  |            ~~~~~~~~~^^^^^^^^^^^^^^^^^
transactions-consumer-1  |   File "/.venv/lib/python3.13/site-packages/click/core.py", line 1078, in main
transactions-consumer-1  |     rv = self.invoke(ctx)
transactions-consumer-1  |   File "/.venv/lib/python3.13/site-packages/click/core.py", line 1688, in invoke
transactions-consumer-1  |     return _process_result(sub_ctx.command.invoke(sub_ctx))
transactions-consumer-1  |                            ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
transactions-consumer-1  |   File "/.venv/lib/python3.13/site-packages/click/core.py", line 1688, in invoke
transactions-consumer-1  |     return _process_result(sub_ctx.command.invoke(sub_ctx))
transactions-consumer-1  |                            ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
transactions-consumer-1  |   File "/.venv/lib/python3.13/site-packages/click/core.py", line 1434, in invoke
transactions-consumer-1  |     return ctx.invoke(self.callback, **ctx.params)
transactions-consumer-1  |            ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
transactions-consumer-1  |   File "/.venv/lib/python3.13/site-packages/click/core.py", line 783, in invoke
transactions-consumer-1  |     return __callback(*args, **kwargs)
transactions-consumer-1  |   File "/.venv/lib/python3.13/site-packages/click/decorators.py", line 33, in new_func
transactions-consumer-1  |     return f(get_current_context(), *args, **kwargs)
transactions-consumer-1  |   File "/usr/src/sentry/src/sentry/runner/decorators.py", line 82, in inner
transactions-consumer-1  |     return ctx.invoke(f, *args, **kwargs)
transactions-consumer-1  |            ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
transactions-consumer-1  |   File "/.venv/lib/python3.13/site-packages/click/core.py", line 783, in invoke
transactions-consumer-1  |     return __callback(*args, **kwargs)
transactions-consumer-1  |   File "/.venv/lib/python3.13/site-packages/click/decorators.py", line 33, in new_func
transactions-consumer-1  |     return f(get_current_context(), *args, **kwargs)
transactions-consumer-1  |   File "/usr/src/sentry/src/sentry/runner/decorators.py", line 34, in inner
transactions-consumer-1  |     return ctx.invoke(f, *args, **kwargs)
transactions-consumer-1  |            ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
transactions-consumer-1  |   File "/.venv/lib/python3.13/site-packages/click/core.py", line 783, in invoke
transactions-consumer-1  |     return __callback(*args, **kwargs)
transactions-consumer-1  |   File "/usr/src/sentry/src/sentry/runner/commands/run.py", line 569, in basic_consumer
transactions-consumer-1  |     run_processor_with_signals(processor, consumer_name)
transactions-consumer-1  |     ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
transactions-consumer-1  |   File "/usr/src/sentry/src/sentry/utils/kafka.py", line 46, in run_processor_with_signals
transactions-consumer-1  |     processor.run()
transactions-consumer-1  |     ~~~~~~~~~~~~~^^
transactions-consumer-1  |   File "/.venv/lib/python3.13/site-packages/arroyo/processing/processor.py", line 333, in run
transactions-consumer-1  |     self._run_once()
transactions-consumer-1  |     ~~~~~~~~~~~~~~^^
transactions-consumer-1  |   File "/.venv/lib/python3.13/site-packages/arroyo/processing/processor.py", line 396, in _run_once
transactions-consumer-1  |     self.__message = self.__consumer.poll(timeout=1.0)
transactions-consumer-1  |                      ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
transactions-consumer-1  |   File "/.venv/lib/python3.13/site-packages/arroyo/backends/kafka/consumer.py", line 422, in poll
transactions-consumer-1  |     raise OffsetOutOfRange(str(error))
transactions-consumer-1  | arroyo.errors.OffsetOutOfRange: KafkaError{code=_AUTO_OFFSET_RESET,val=-140,str="fetch failed due to requested offset not available on the broker: Broker: Offset out of range (broker 1001)"}
transactions-consumer-1  | Updating certificates in /etc/ssl/certs...
transactions-consumer-1  | 0 added, 0 removed; done.

GoldUser32 avatar May 03 '25 12:05 GoldUser32

Sentry 25.4.0

We were having similar issues (no new transactions appearing) after recovering from our single-host deployment getting overwhelmed (note to self, 100% sampling on services that process thousands of messages a minute is a bad idea).

After trying some things mentioned in other issues like https://github.com/getsentry/self-hosted/issues/3138 and https://github.com/getsentry/self-hosted/issues/2876, I resolved it by taking down the instance, removing the two kafka volumes, then re-running ./install.sh and docker compose up.

Complete steps were something like:

docker compose down
docker volume prune
<some combination of docker ps -a, docker stop, and docker rm to remove the containers using the two kafka volumes>
docker volume rm sentry-kafka
docker volume rm sentry-self-hosted_sentry-kafka-log
./install.sh
docker compose --env-file .env --env-file .env.custom up -d

Left it running for an hour and when I next checked, the live/recent transactions were appearing again.

Though - they're not showing in Discover, but they do show in Traces. Interesting, will have to keep digging I suppose.

lettucemode avatar May 14 '25 19:05 lettucemode

@GoldUser32 are you still experiencing the issue? The Kafka offset out of range issue means you'll need to reset the offset for that topic (or easy enough, you can reset all topic's offset) and restart the containers.

aldy505 avatar Jun 26 '25 04:06 aldy505

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

getsantry[bot] avatar Jul 17 '25 07:07 getsantry[bot]

Same issue here we had a traffic spike, and afterward, some containers failed their health checks with the following error:

KafkaError{code=_AUTO_OFFSET_RESET,val=-140,str="fetch failed due to requested offset not available on the broker: Broker: Offset out of range (broker 1001)"}

I was able to recover with:

docker compose stop
docker compose start kafka
docker compose exec -it kafka bash

Then inside the Kafka container, I ran this to reset consumer group offsets:

for group in $(/usr/bin/kafka-consumer-groups --bootstrap-server localhost:9092 --list); do echo "Resetting $group"; /usr/bin/kafka-consumer-groups --bootstrap-server localhost:9092 --group "$group" --reset-offsets --all-topics --to-latest --execute; done

maantje avatar Jul 18 '25 22:07 maantje

@maantje there is a solution for your problem on the docs: https://develop.sentry.dev/self-hosted/troubleshooting/kafka/#offset-out-of-range-error

aldy505 avatar Jul 20 '25 00:07 aldy505

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

getsantry[bot] avatar Aug 10 '25 07:08 getsantry[bot]