self-hosted
self-hosted copied to clipboard
getsentry
High CPU and RAM consumption
Self-Hosted Version
22.9.0 (on Ubuntu 22.04)
CPU Architecture
x86_64
Docker Version
20.10.18, build b40c2f6
Docker Compose Version
2.10.2
Steps to Reproduce
Install Sentry-self hosted, leave it running for a while.
Expected Result
Normal CPU and RAM consumption.
Actual Result
When I first installed Sentry self-hosted, and started sending it data, all was going well. The self-hosted dashboard was reporting performance and error data as expected, and RAM/CPU usage was completely under control.
But then overnight the CPU and RAM usage on the server went crazy, with CPU maxing out, high RAM and swap usage. This appeared to be down to sentry, because running docker compose down restored things back to normal (except that sentry was no longer running).
This seems very similar to this issue from @umgbhalla but there wasn't any apparent resolution there, except maybe to re-install sentry from fresh. I also have 32 GB RAM and 8 cores. It's also similar to this issue from @wodCZ, but I'm not seeing the little CPU spikes.
I tried installing sentry from fresh, but CPU was still maxing out after:
The short period maxing out at 100% was when I set it running again, followed soon after by shutting it down again.
However, I don't think that it was a completely fresh install, because despite removing the self-hosted folder completely and re-installing, when running docker compose run --rm web createuser after installing to add a user (I install with ./install.sh --skip-user-creation to avoid any prompts during installation), it said User: [email protected] exists, use --force-update to force... so it still had some retained state from before?
In any case, it shouldn't really be necessary to re-install in order to get CPU/RAM back under control. Any ideas?
(And an aside... is it possible to script the docker compose run --rm web createuser step, i.e. with no user prompts and instead pass the info by args or env vars?)
I remember there were some issues after upgrades, which caused a process to hang in restart loop due to some incompatibility.
You could see if some service keeps spamming the docker compose logs -f output.
Dangerous, you will lose data: For a fresh install you might need to drop the volumes too (docker compose down -v), then check that there's no leftover sentry volume using docker system df -v.
Checking which container is causing a high CPU usage could be helpful too: docker stats -a
@wodCZ I really appreciate the reply.
There does seem to be something spamming the docker compose logs -f output... small sample below...
sentry-self-hosted-post-process-forwarder-1 | 11:43:51 [INFO] batching-kafka-consumer: Worker flush took 5ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:51 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [64993, 64993]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:51 [INFO] batching-kafka-consumer: Worker flush took 10ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:51 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [64994, 64994]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:51 [INFO] batching-kafka-consumer: Worker flush took 8ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:51 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [64995, 64995]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:51 [INFO] batching-kafka-consumer: Worker flush took 9ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:51 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [64996, 64996]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:51 [INFO] batching-kafka-consumer: Worker flush took 5ms
sentry-self-hosted-snuba-transactions-consumer-1 | 2022-09-23 11:43:51,295 Completed processing <Batch: 1 message, open for 0.82 seconds>.
sentry-self-hosted-clickhouse-1 | 2022.09.23 11:43:51.509446 [ 95 ] {0a61de27-eaae-4301-b955-3a29f9b5fc6b} <Information> executeQuery: Read 19 rows, 72.86 KiB in 0.020 sec., 938 rows/sec., 3.51 MiB/sec.
sentry-self-hosted-clickhouse-1 | 2022.09.23 11:43:51.512513 [ 95 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-clickhouse-1 | 2022.09.23 11:43:51.529926 [ 96 ] {1603a3d4-a4d0-4ba1-a304-c673f0dce9b4} <Information> executeQuery: Read 19 rows, 1.00 KiB in 0.018 sec., 1040 rows/sec., 54.80 KiB/sec.
sentry-self-hosted-clickhouse-1 | 2022.09.23 11:43:51.537604 [ 96 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-snuba-outcomes-consumer-1 | 2022-09-23 11:43:51,540 Completed processing <Batch: 19 messages, open for 0.83 seconds>.
sentry-self-hosted-snuba-consumer-1 | 2022-09-23 11:43:51,514 Completed processing <Batch: 19 messages, open for 0.80 seconds>.
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [64997, 64997]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 4ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [64998, 64998]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 3ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [64999, 64999]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 9ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65000, 65000]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 6ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65001, 65001]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 3ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65002, 65002]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 5ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65003, 65003]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 4ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65004, 65004]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 3ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65005, 65005]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 3ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65006, 65006]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 6ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65007, 65007]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 10ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65008, 65008]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 6ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65009, 65009]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 3ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65010, 65010]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 3ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65011, 65011]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 5ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65012, 65012]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 4ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65013, 65013]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 3ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65014, 65014]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 4ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65015, 65015]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:52 [INFO] batching-kafka-consumer: Worker flush took 4ms
sentry-self-hosted-clickhouse-1 | 2022.09.23 11:43:52.316671 [ 96 ] {b3256d45-320f-42e0-88be-8a73868e6844} <Information> executeQuery: Read 17 rows, 919.00 B in 0.022 sec., 762 rows/sec., 40.26 KiB/sec.
sentry-self-hosted-clickhouse-1 | 2022.09.23 11:43:52.321504 [ 95 ] {873acf29-5f64-4c92-a6fe-d6495dd41239} <Information> executeQuery: Read 17 rows, 65.73 KiB in 0.027 sec., 636 rows/sec., 2.40 MiB/sec.
sentry-self-hosted-clickhouse-1 | 2022.09.23 11:43:52.324163 [ 95 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-clickhouse-1 | 2022.09.23 11:43:52.326841 [ 96 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-snuba-outcomes-consumer-1 | 2022-09-23 11:43:52,328 Completed processing <Batch: 17 messages, open for 0.82 seconds>.
sentry-self-hosted-snuba-consumer-1 | 2022-09-23 11:43:52,328 Completed processing <Batch: 17 messages, open for 0.84 seconds>.
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65016, 65016]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 3ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65017, 65017]}): forced:False size:True time:False
sentry-self-hosted-clickhouse-1 | 2022.09.23 11:43:53.070237 [ 96 ] {26cccaf0-7fb3-4037-8b9f-a7e63b1902cd} <Information> executeQuery: Read 16 rows, 866.00 B in 0.006 sec., 2649 rows/sec., 140.02 KiB/sec.
sentry-self-hosted-clickhouse-1 | 2022.09.23 11:43:53.070653 [ 96 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-clickhouse-1 | 2022.09.23 11:43:53.082137 [ 95 ] {2655494c-39d8-4ed9-ae56-5d1fb5960562} <Information> executeQuery: Read 16 rows, 56.80 KiB in 0.018 sec., 877 rows/sec., 3.04 MiB/sec.
sentry-self-hosted-clickhouse-1 | 2022.09.23 11:43:53.082556 [ 95 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-snuba-outcomes-consumer-1 | 2022-09-23 11:43:53,073 Completed processing <Batch: 16 messages, open for 0.78 seconds>.
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 11ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65018, 65018]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 4ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65019, 65019]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 10ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65020, 65020]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 3ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65021, 65021]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 4ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65022, 65022]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 5ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65023, 65023]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 6ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65024, 65024]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 5ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65025, 65025]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 3ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65026, 65026]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 5ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65027, 65027]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 5ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65028, 65028]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 6ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65029, 65029]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 4ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65030, 65030]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 6ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65031, 65031]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 5ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65032, 65032]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 6ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65033, 65033]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 12ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65034, 65034]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 8ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65035, 65035]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 5ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65036, 65036]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 5ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65037, 65037]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 9ms
sentry-self-hosted-snuba-consumer-1 | 2022-09-23 11:43:53,084 Completed processing <Batch: 16 messages, open for 0.79 seconds>.
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65038, 65038]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 4ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65039, 65039]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 8ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65040, 65040]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 7ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65041, 65041]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 6ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65042, 65042]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 5ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65043, 65043]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 6ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65044, 65044]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 5ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65045, 65045]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 3ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65046, 65046]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 3ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65047, 65047]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 4ms
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {('events', 0): [65048, 65048]}): forced:False size:True time:False
sentry-self-hosted-post-process-forwarder-1 | 11:43:53 [INFO] batching-kafka-consumer: Worker flush took 3ms
As for docker stats -a, there are a few major CPU hogs:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
c2b394f89938 sentry-self-hosted-nginx-1 0.00% 7.332MiB / 30.61GiB 0.02% 1.38kB / 0B 5.65MB / 4.1kB 9
f145b7803578 sentry-self-hosted-relay-1 0.31% 34.12MiB / 30.61GiB 0.11% 6.97kB / 6.37kB 12MB / 0B 38
c1249a215531 sentry-self-hosted-worker-1 233.27% 874.8MiB / 30.61GiB 2.79% 830MB / 1.03GB 6.16MB / 34.9MB 68
710a423a3849 sentry-self-hosted-sentry-cleanup-1 0.00% 1.16MiB / 30.61GiB 0.00% 3.29kB / 0B 164kB / 217kB 1
772c5d6bc99d sentry-self-hosted-snuba-cleanup-1 0.00% 13.21MiB / 30.61GiB 0.04% 4.81kB / 1.97kB 11.4MB / 143kB 1
597010fda5ed sentry-self-hosted-web-1 0.01% 415.8MiB / 30.61GiB 1.33% 34.2kB / 33.4kB 6.05MB / 270kB 19
073198558a64 sentry-self-hosted-subscription-consumer-transactions-1 0.23% 123.4MiB / 30.61GiB 0.39% 215kB / 143kB 36.9kB / 205kB 6
3b0ef276d52f sentry-self-hosted-ingest-consumer-1 0.24% 128.3MiB / 30.61GiB 0.41% 381kB / 200kB 32.8kB / 205kB 6
923052b6610c sentry-self-hosted-snuba-transactions-cleanup-1 0.00% 10.26MiB / 30.61GiB 0.03% 4.81kB / 1.98kB 8.27MB / 131kB 1
15f98ab07e28 sentry-self-hosted-subscription-consumer-events-1 0.29% 123.8MiB / 30.61GiB 0.39% 213kB / 141kB 508kB / 201kB 6
88164011a5ec sentry-self-hosted-post-process-forwarder-1 12.64% 136.1MiB / 30.61GiB 0.43% 350MB / 16.6MB 16.4kB / 205kB 13
457087d09979 sentry-self-hosted-cron-1 0.01% 148.7MiB / 30.61GiB 0.47% 32kB / 155kB 819kB / 270kB 3
6712f96c98c3 sentry-self-hosted-geoipupdate-1 0.00% 0B / 0B 0.00% 0B / 0B 0B / 0B 0
6ee727622535 sentry-self-hosted-symbolicator-cleanup-1 0.00% 1.559MiB / 30.61GiB 0.00% 3.6kB / 0B 938kB / 12.3kB 1
722467cd7c17 sentry-self-hosted-postgres-1 57.74% 98.79MiB / 30.61GiB 0.32% 107MB / 47MB 12MB / 133MB 20
87e1ab6a15a5 sentry-self-hosted-snuba-transactions-consumer-1 0.84% 80.23MiB / 30.61GiB 0.26% 225MB / 1.6MB 827kB / 0B 9
59efe315bd55 sentry-self-hosted-memcached-1 6.72% 16.31MiB / 30.61GiB 0.05% 36.3MB / 84.6MB 995kB / 41kB 10
09f64b0d0348 sentry-self-hosted-snuba-api-1 0.01% 85.38MiB / 30.61GiB 0.27% 15.8kB / 33.2kB 5.02MB / 65.5kB 7
792a3c51352e sentry-self-hosted-snuba-subscription-consumer-transactions-1 0.47% 78.24MiB / 30.61GiB 0.25% 339kB / 178kB 205kB / 0B 9
86955e5f1c57 sentry-self-hosted-smtp-1 0.00% 1.621MiB / 30.61GiB 0.01% 3.67kB / 0B 258kB / 4.1kB 2
286670644c63 sentry-self-hosted-snuba-outcomes-consumer-1 1.48% 78.29MiB / 30.61GiB 0.25% 3.59MB / 2.21MB 73.7kB / 0B 10
ea23c1a0e793 sentry-self-hosted-snuba-subscription-consumer-events-1 0.37% 78.23MiB / 30.61GiB 0.25% 340kB / 177kB 160kB / 0B 9
0255012d5077 sentry-self-hosted-snuba-replacer-1 0.27% 77.82MiB / 30.61GiB 0.25% 206kB / 134kB 20.5kB / 0B 5
d5872f0aab61 sentry-self-hosted-snuba-consumer-1 2.88% 79.8MiB / 30.61GiB 0.25% 219MB / 20MB 553kB / 0B 10
64e10abfea02 sentry-self-hosted-snuba-sessions-consumer-1 0.31% 78.05MiB / 30.61GiB 0.25% 203kB / 131kB 0B / 0B 8
1c36a15d6a76 sentry-self-hosted-symbolicator-1 0.01% 9.203MiB / 30.61GiB 0.03% 3.67kB / 0B 2.2MB / 0B 26
2d115aeef1c2 sentry-self-hosted-kafka-1 6.81% 608.7MiB / 30.61GiB 1.94% 227MB / 798MB 3.54MB / 212MB 83
e4681ebf6c6d sentry-self-hosted-zookeeper-1 0.14% 78.37MiB / 30.61GiB 0.25% 43.6kB / 49.1kB 3.13MB / 422kB 67
354b3064f3e8 sentry-self-hosted-redis-1 8.96% 113.2MiB / 30.61GiB 0.36% 683MB / 700MB 6.25MB / 219MB 5
5b5d617cbe77 sentry-self-hosted-clickhouse-1 17.80% 841.7MiB / 30.61GiB 2.68% 19.3MB / 458kB 14.4MB / 822MB 62
Note that this server isn't dedicated to sentry... there are some of my own containers running, the worst of which is probably this one:
d29a327d8ceb srv-hsjdkfk--jdheudnfu.2.od2sau9y2xkun6qt9olpkh719 272.72% 1.551GiB / 30.61GiB 5.07% 214MB / 641MB 65.3MB / 23.9MB 129
And there does seem to be quite a bit left over even after running docker compose down -v:
~/sentry/self-hosted# docker compose down -v
[+] Running 7/0
⠿ Volume sentry-self-hosted_sentry-secrets Removed 0.0s
⠿ Volume sentry-self-hosted_sentry-nginx-cache Removed 0.0s
⠿ Volume sentry-self-hosted_sentry-kafka-log Removed 0.0s
⠿ Volume sentry-self-hosted_sentry-clickhouse-log Removed 0.0s
⠿ Volume sentry-self-hosted_sentry-smtp-log Removed 0.0s
⠿ Volume sentry-self-hosted_sentry-smtp Removed 0.0s
⠿ Volume sentry-self-hosted_sentry-zookeeper-log Removed 0.0s
~/sentry/self-hosted# docker system df -v | grep sentry
sentry-cleanup-self-hosted-local latest e7ede052ba7f 34 minutes ago 848.2MB 846.9MB 1.386MB 0
sentry-self-hosted-local latest ca31283ff33a 34 minutes ago 846.9MB 846.9MB 0B 0
getsentry/relay nightly f9338e4e54e9 2 hours ago 238.7MB 85.19MB 153.5MB 0
getsentry/sentry nightly 7e6a40d43a81 2 hours ago 846.8MB 846.8MB 0B 0
getsentry/snuba nightly 97d532d10472 15 hours ago 427.7MB 427.7MB 0B 0
getsentry/symbolicator nightly 9ad1d9ba1684 3 days ago 151.3MB 151.3MB 0B 0
sentry-clickhouse 0 20.96MB
sentry-data 0 0B
sentry-symbolicator 0 0B
sentry-zookeeper 0 341.4kB
sentry-kafka 0 7.178GB
sentry-postgres 0 766.9MB
sentry-redis 0 30.24MB
I cleared out all unused volumes with docker volume prune, then did a fresh install, and this time indeed it didn't have any memory of any previous user.
And sure enough, now when I start sentry, the docker compose logs -f output is completely clean before I start sending data, and CPU is completely fine even once I start sending data from my production system, with a nice, clean log showing that it's just calming receiving and processing data:
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:40:04.028748 [ 927 ] {6ec623a5-39b4-451f-a2cd-e6c4ec170f0b} <Information> executeQuery: Read 3 rows, 1.45 KiB in 0.002 sec., 1338 rows/sec., 646.28 KiB/sec.
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:40:04.028984 [ 927 ] {} <Information> TCPHandler: Processed in 0.003 sec.
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:40:04.182565 [ 928 ] {} <Information> TCPHandler: Done processing connection.
sentry-self-hosted-snuba-transactions-cleanup-1 | 2022-09-23 13:40:04,030 Dropped 0 partitions on clickhouse:9000
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:40:04.219153 [ 927 ] {} <Information> TCPHandler: Done processing connection.
sentry-self-hosted-nginx-1 | 2022/09/23 13:40:11 [warn] 23#23: *277 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000018, client: 10.0.1.32, server: , request: "POST /api/1/envelope/?sentry_key=78e11857f9ef524006826a7f34fdb564&sentry_version=7&sentry_client=sentry.javascript.node%2F7.13.0 HTTP/1.1", host: "sentry.myserver.com"
sentry-self-hosted-nginx-1 | 10.0.1.32 - - [23/Sep/2022:13:40:11 +0000] "POST /api/1/envelope/?sentry_key=78e11857f9ef524006826a7f34fdb564&sentry_version=7&sentry_client=sentry.javascript.node%2F7.13.0 HTTP/1.1" 200 41 "-" "-" "172.18.0.1, 10.0.1.5"
sentry-self-hosted-nginx-1 | 2022/09/23 13:40:12 [warn] 23#23: *279 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000019, client: 10.0.1.32, server: , request: "POST /api/1/envelope/?sentry_key=78e11857f9ef524006826a7f34fdb564&sentry_version=7&sentry_client=sentry.javascript.node%2F7.13.0 HTTP/1.1", host: "sentry.myserver.com"
sentry-self-hosted-nginx-1 | 10.0.1.32 - - [23/Sep/2022:13:40:12 +0000] "POST /api/1/envelope/?sentry_key=78e11857f9ef524006826a7f34fdb564&sentry_version=7&sentry_client=sentry.javascript.node%2F7.13.0 HTTP/1.1" 200 41 "-" "-" "95.217.18.255, 10.0.1.5"
sentry-self-hosted-ingest-consumer-1 | 13:40:13 [INFO] batching-kafka-consumer: Flushing 2 items (from {('ingest-transactions', 0): [24, 25]}): forced:False size:False time:True
sentry-self-hosted-ingest-consumer-1 | 13:40:13 [INFO] batching-kafka-consumer: Worker flush took 43ms
sentry-self-hosted-snuba-transactions-consumer-1 | 2022-09-23 13:40:14,597 Completed processing <Batch: 2 messages, open for 1.05 seconds>.
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:40:14.576825 [ 86 ] {52a7981a-e29b-4d3d-874e-3bea4b56a768} <Information> executeQuery: Read 2 rows, 124.00 B in 0.013 sec., 153 rows/sec., 9.27 KiB/sec.
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:40:14.577499 [ 86 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:40:14.590924 [ 871 ] {0321dcfc-8f99-418b-81b8-38aee257a35e} <Information> executeQuery: Read 2 rows, 3.85 KiB in 0.024 sec., 81 rows/sec., 157.42 KiB/sec.
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:40:14.591532 [ 871 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-snuba-outcomes-consumer-1 | 2022-09-23 13:40:14,583 Completed processing <Batch: 2 messages, open for 1.03 seconds>.
sentry-self-hosted-nginx-1 | 2022/09/23 13:40:19 [warn] 23#23: *281 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000020, client: 10.0.1.32, server: , request: "POST /api/1/envelope/?sentry_key=78e11857f9ef524006826a7f34fdb564&sentry_version=7&sentry_client=sentry.javascript.node%2F7.13.0 HTTP/1.1", host: "sentry.myserver.com"
sentry-self-hosted-nginx-1 | 10.0.1.32 - - [23/Sep/2022:13:40:19 +0000] "POST /api/1/envelope/?sentry_key=78e11857f9ef524006826a7f34fdb564&sentry_version=7&sentry_client=sentry.javascript.node%2F7.13.0 HTTP/1.1" 200 41 "-" "-" "95.217.18.255, 10.0.1.5"
sentry-self-hosted-ingest-consumer-1 | 13:40:20 [INFO] batching-kafka-consumer: Flushing 1 items (from {('ingest-transactions', 0): [26, 26]}): forced:False size:False time:True
sentry-self-hosted-ingest-consumer-1 | 13:40:20 [INFO] batching-kafka-consumer: Worker flush took 15ms
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:40:21.427031 [ 871 ] {429e4f5e-8b43-450f-813c-b988ece75c4f} <Information> executeQuery: Read 1 rows, 71.00 B in 0.011 sec., 94 rows/sec., 6.55 KiB/sec.
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:40:21.427716 [ 871 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:40:21.433547 [ 86 ] {ba60fced-5f24-4cfd-81c7-cdf5000c5fb8} <Information> executeQuery: Read 1 rows, 2.04 KiB in 0.021 sec., 48 rows/sec., 99.37 KiB/sec.
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:40:21.434016 [ 86 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-snuba-outcomes-consumer-1 | 2022-09-23 13:40:21,431 Completed processing <Batch: 1 message, open for 1.02 seconds>.
sentry-self-hosted-snuba-transactions-consumer-1 | 2022-09-23 13:40:21,437 Completed processing <Batch: 1 message, open for 1.03 seconds>.
sentry-self-hosted-nginx-1 | 2022/09/23 13:40:28 [warn] 23#23: *283 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000021, client: 10.0.1.32, server: , request: "POST /api/1/envelope/?sentry_key=78e11857f9ef524006826a7f34fdb564&sentry_version=7&sentry_client=sentry.javascript.node%2F7.13.0 HTTP/1.1", host: "sentry.myserver.com"
sentry-self-hosted-nginx-1 | 10.0.1.32 - - [23/Sep/2022:13:40:28 +0000] "POST /api/1/envelope/?sentry_key=78e11857f9ef524006826a7f34fdb564&sentry_version=7&sentry_client=sentry.javascript.node%2F7.13.0 HTTP/1.1" 200 41 "-" "-" "172.18.0.1, 10.0.1.5"
sentry-self-hosted-ingest-consumer-1 | 13:40:29 [INFO] batching-kafka-consumer: Flushing 1 items (from {('ingest-transactions', 0): [27, 27]}): forced:False size:False time:True
sentry-self-hosted-ingest-consumer-1 | 13:40:29 [INFO] batching-kafka-consumer: Worker flush took 11ms
sentry-self-hosted-snuba-transactions-consumer-1 | 2022-09-23 13:40:30,582 Completed processing <Batch: 1 message, open for 1.03 seconds>.
sentry-self-hosted-snuba-outcomes-consumer-1 | 2022-09-23 13:40:30,567 Completed processing <Batch: 1 message, open for 1.01 seconds>.
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:40:30.564002 [ 86 ] {28b6de30-aa77-45b3-96ec-d95d8e3b4f09} <Information> executeQuery: Read 1 rows, 71.00 B in 0.005 sec., 183 rows/sec., 12.75 KiB/sec.
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:40:30.565143 [ 86 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:40:30.574041 [ 88 ] {6c1e39a1-b496-46c1-b34c-e42f0689f6c0} <Information> executeQuery: Read 1 rows, 2.08 KiB in 0.015 sec., 68 rows/sec., 142.74 KiB/sec.
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:40:30.574501 [ 88 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-nginx-1 | 10.0.1.32 - - [23/Sep/2022:13:40:49 +0000] "POST /api/1/envelope/?sentry_key=78e11857f9ef524006826a7f34fdb564&sentry_version=7&sentry_client=sentry.javascript.node%2F7.13.0 HTTP/1.1" 200 41 "-" "-" "172.18.0.1, 10.0.1.5"
sentry-self-hosted-ingest-consumer-1 | 13:40:50 [INFO] batching-kafka-consumer: Flushing 1 items (from {('ingest-transactions', 0): [28, 28]}): forced:False size:False time:True
sentry-self-hosted-ingest-consumer-1 | 13:40:50 [INFO] batching-kafka-consumer: Worker flush took 19ms
sentry-self-hosted-nginx-1 | 2022/09/23 13:40:50 [warn] 23#23: *287 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000022, client: 10.0.1.32, server: , request: "POST /api/1/envelope/?sentry_key=78e11857f9ef524006826a7f34fdb564&sentry_version=7&sentry_client=sentry.javascript.node%2F7.13.0 HTTP/1.1", host: "sentry.myserver.com"
sentry-self-hosted-nginx-1 | 10.0.1.32 - - [23/Sep/2022:13:40:50 +0000] "POST /api/1/envelope/?sentry_key=78e11857f9ef524006826a7f34fdb564&sentry_version=7&sentry_client=sentry.javascript.node%2F7.13.0 HTTP/1.1" 200 41 "-" "-" "95.217.18.255, 10.0.1.5"
sentry-self-hosted-snuba-transactions-consumer-1 | 2022-09-23 13:40:51,278 Completed processing <Batch: 1 message, open for 1.04 seconds>.
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:40:51.252905 [ 871 ] {cc51e6dc-e066-452e-a2d8-2073d4a57fda} <Information> executeQuery: Read 1 rows, 71.00 B in 0.011 sec., 92 rows/sec., 6.40 KiB/sec.
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:40:51.253389 [ 871 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:40:51.274097 [ 86 ] {6e07f3e6-4d94-4e83-9779-48695a9d3157} <Information> executeQuery: Read 1 rows, 2.14 KiB in 0.028 sec., 36 rows/sec., 77.68 KiB/sec.
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:40:51.274580 [ 86 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-snuba-outcomes-consumer-1 | 2022-09-23 13:40:51,256 Completed processing <Batch: 1 message, open for 1.02 seconds>.
sentry-self-hosted-ingest-consumer-1 | 13:40:51 [INFO] batching-kafka-consumer: Flushing 1 items (from {('ingest-transactions', 0): [29, 29]}): forced:False size:False time:True
sentry-self-hosted-ingest-consumer-1 | 13:40:51 [INFO] batching-kafka-consumer: Worker flush took 20ms
sentry-self-hosted-snuba-outcomes-consumer-1 | 2022-09-23 13:40:52,572 Completed processing <Batch: 1 message, open for 1.02 seconds>.
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:40:52.567509 [ 871 ] {cd393bc3-29e9-4554-ad3b-9bd370ed12bc} <Information> executeQuery: Read 1 rows, 71.00 B in 0.009 sec., 117 rows/sec., 8.12 KiB/sec.
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:40:52.568424 [ 871 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:40:52.573708 [ 86 ] {a963468b-41f3-4b0b-b5e1-e99b95f9ae6c} <Information> executeQuery: Read 1 rows, 2.05 KiB in 0.016 sec., 64 rows/sec., 131.65 KiB/sec.
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:40:52.573999 [ 86 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-snuba-transactions-consumer-1 | 2022-09-23 13:40:52,578 Completed processing <Batch: 1 message, open for 1.02 seconds>.
sentry-self-hosted-nginx-1 | 2022/09/23 13:41:02 [warn] 23#23: *289 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000023, client: 10.0.1.32, server: , request: "POST /api/1/envelope/?sentry_key=78e11857f9ef524006826a7f34fdb564&sentry_version=7&sentry_client=sentry.javascript.node%2F7.13.0 HTTP/1.1", host: "sentry.myserver.com"
sentry-self-hosted-nginx-1 | 10.0.1.32 - - [23/Sep/2022:13:41:02 +0000] "POST /api/1/envelope/?sentry_key=78e11857f9ef524006826a7f34fdb564&sentry_version=7&sentry_client=sentry.javascript.node%2F7.13.0 HTTP/1.1" 200 41 "-" "-" "95.217.18.255, 10.0.1.5"
sentry-self-hosted-ingest-consumer-1 | 13:41:03 [INFO] batching-kafka-consumer: Flushing 1 items (from {('ingest-transactions', 0): [30, 30]}): forced:False size:False time:True
sentry-self-hosted-ingest-consumer-1 | 13:41:03 [INFO] batching-kafka-consumer: Worker flush took 20ms
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:41:04.212943 [ 88 ] {85e86ebc-d71b-48ba-8a45-be7c1e263815} <Information> executeQuery: Read 1 rows, 71.00 B in 0.012 sec., 83 rows/sec., 5.81 KiB/sec.
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:41:04.213436 [ 88 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:41:04.225864 [ 871 ] {63aebc9c-4066-4d7d-94d2-f7b6ea4e7829} <Information> executeQuery: Read 1 rows, 2.00 KiB in 0.023 sec., 43 rows/sec., 86.27 KiB/sec.
sentry-self-hosted-clickhouse-1 | 2022.09.23 13:41:04.226333 [ 871 ] {} <Information> HTTPHandler: Done processing query
sentry-self-hosted-snuba-outcomes-consumer-1 | 2022-09-23 13:41:04,218 Completed processing <Batch: 1 message, open for 1.02 seconds>.
sentry-self-hosted-snuba-transactions-consumer-1 | 2022-09-23 13:41:04,232 Completed processing <Batch: 1 message, open for 1.04 seconds>.
So it seems that sentry somehow gets stuck in some horrible loop (as @wodCZ suspected), with log output as per the post above... causing CPU to max out and the whole thing grinds to a halt... any ideas?
Thanks for bringing this up to us...looks like this will require some additional investigation. I'll keep this thread updated if I can find something meaningful
@drmrbrewer How many requests per second are you handling at your self-hosted instance at the time of cpu spike?
I mean maybe there is another reason which sentry keeps getting hit more and more and therefore higher cpu usage.
@aminvakil it's a fair question but I don't think it could just be due to a surge in traffic. Traffic is normally around 5 requests per second, and wouldn't go above around 15 per second, but I have actually set tracesSampleRate to 0.01 so Sentry is normally pretty quiet. This does seem like something triggers Sentry to go haywire. I should add that, since re-installing Sentry, I haven't seen this again, so it is probably quite rare?
@drmrbrewer Sorry for late response.
It's good to see your issue has got solved, but I had not seen this in my experience prior to your instance, but as reinstalling self-hosted has fixed the issue, I think maybe there was a problem with installation at the first place, not sure what though.
I experienced a similar issue recently: Sentry consumed all available RAM and caused system to cripple with endless OOM-killer events. Restarting Sentry helped for several hours before the issue came back. From docker container stats output it was clear that Redis is the problem. The solution was to change Redis configuration, we added two lines in Sentry startup script:
docker compose exec redis redis-cli CONFIG SET maxmemory-policy volatile-ttl
docker compose exec redis redis-cli CONFIG SET maxmemory 4G
@afaianswq interesting... I don't know enough about redis to understand how far this changes the redis setup from its default? Is this setup something that the Sentry team could safely add to the default Sentry startup script for all?
I am no expert either, I just came across this article when searched for solution: https://yuretspro.medium.com/on-premise-sentry-under-the-high-load-how-to-stay-up-running-be6a0a33791f
I have also experienced this yesterday. Here is my story. :)
./reset.sh script did not help, however this article provided some great info and helped me in a cleanup process: https://yuretspro.medium.com/on-premise-sentry-under-the-high-load-how-to-stay-up-running-be6a0a33791f
REDIS was the issue using up 100% of RAM (8GB) on my server, kafka was also full of events, ingest-events and ingest-transactions. Purged them all as described in that article.
However after cleaning it out and wiping out kafka entries it was still on something... After setting maxmemory-policy to volatile-ttl and maxmemory to 1G it stayed at at 6GB. I was exhausted and p***ed off having to fight this issue entire day so I went: https://www.youtube.com/watch?v=HjPgdhRsGIQ&ab_channel=MasoodBoomgaard
Today is a nother day as he says... :) I came back to my office prepared to wage a mass scale war against him for yet another day.! However..., for my pleasant surprise,... he is back to normal! He is perfectly healthy and no longer fighting to it's own toughs and he even remembered what we have talked about yesterday....??? Whaat?? I tried sending some errors to it after full wipe out, bit it was still offended and didn't wanted to do what it's supposed to do, that is to report these errors as they appear... So...what the ***???? :)
Anyways... until there is no remedy for these premenstrual syndromes of self hosted instance... can at least anybody offer a way to put boundaries on it? So that it would not be allowed to over explode on it's own. Like limiting amount of events he can accept, amount of RAM, amount of storage it can use, etc..?
Also, how do you permanently set these settings for redis (damn... I hate docker...)?
P.S. I am not rude, I am just sarcastic. :) Also indefinitely grateful for everybody who participated in creating this life changing tool! :)
Ok, I have also figured out on how to set permanent redis config to our docker whale. :)
One must edit docker-compose.yaml file like this:
redis:
<<: *restart_policy
image: "redis:6.2.10-alpine"
command: redis-server /usr/local/etc/redis/redis.conf
healthcheck:
<<: *healthcheck_defaults
test: redis-cli ping
volumes:
- "sentry-redis:/data"
- "/root/redis.conf:/usr/local/etc/redis/redis.conf"
ulimits:
nofile:
soft: 10032
hard: 10032
This tells a whale to run redis in the docker container with command, which also explicitly defines redis.conf file. Otherwise redis is running with it's own internal config. Then one must grab example redis.conf file from redis github repository https://raw.githubusercontent.com/redis/redis/6.2/redis.conf and put it in to /root/redis.conf
Note that it must match redis image version!!! Otherwise docker-compose fails with totally misleading error: "ERROR: for snuba-consumer Container "16a56c6c0df5" is unhealthy."
Now /root/config.file can be edited and redis settings modified accordingly. I have added these mods:
bind * -::* # this is a dedault value of docerized redis instance, must be replicated
protected-mode no # same as above
maxmemory-policy volatile-ttl
maxmemory 1G
save ""
Also if you one experiences errors after modifying redis settings, I found out these two commands very useful:
docker exec -i sentry-self-hosted_redis_1 redis-cli info - shows actual config of running dockerized redis instance. If it fails to show it, it means redis was not started, so something is wrong with your reis.conf file.
docker logs sentry-self-hosted_redis_1 - shows log output of redis instance.
I believe most of us are developers, instead of dev-ops monkeys... so figuring things like these takes out a lot of time, struggle and effort. It would be much better if that self hosted instance would have this already included with redis.conf file included in self-hosted repository root also included on .gitignore file, so that users would also be able tweak other redis settings if necessary. Same could be done with other services accordingly.
Cheers!
Like limiting amount of events he can accept
@Simanas we've configured rate limit on our Sentry instance, and I didn't need to "wage a mass scale war against him" even with several not so lucky releases on our apps.
You can find it in admin settings (/manage/settings/)
Also, if you have a single problematic project which occasionally bursts into fire, you can rate-limit just the project in Project settings -> Client Keys (SDK) -> Configure.