sentry not receiving error from apps after 1 day install.

[addityar]

Version

21.7.0

Steps to Reproduce

hello,

I have problem with our sentry, 1 day ago everything is normal, sentry can show error from our apps. then, today sentry not receiving any error from our apps.

i see the logs many problem with internal docker connection. how to solve this case ?

Expected Result

no error

Actual Result

relay log (docker-compose logs relay ) 2022-07-07T05:08:44Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream caused by: error sending request for url (http://web:9000/api/0/relays/register/challenge/): error trying to connect: tcp connect error: Connection refused (os error 111) 2022-07-07T05:08:44Z [relay_server::actors::upstream] WARN: Network outage, scheduling another check in 1s 2022-07-07T05:08:45Z [relay_server::actors::upstream] WARN: Network outage, scheduling another check in 1.5s 2022-07-07T05:08:47Z [relay_server::actors::upstream] WARN: Network outage, scheduling another check in 2.25s 2022-07-07T05:08:49Z [relay_server::actors::upstream] WARN: Network outage, scheduling another check in 3.375s 2022-07-07T05:08:52Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream caused by: error sending request for url (http://web:9000/api/0/relays/register/challenge/): error trying to connect: tcp connect error: Connection refused (os error 111) 2022-07-07T05:08:52Z [relay_server::actors::upstream] WARN: Network outage, scheduling another check in 5.0625s 2022-07-07T05:08:57Z [relay_server::actors::upstream] WARN: Network outage, scheduling another check in 7.59375s 2022-07-07T05:09:02Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.20.0.29:9092 failed: Connection refused (after 0ms in state CONNECT, 30 identical error(s) suppressed) 2022-07-07T05:09:02Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.20.0.29:9092 failed: Connection refused (after 0ms in state CONNECT, 30 identical error(s) suppressed) 2022-07-07T05:09:02Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.20.0.29:9092 failed: Connection refused (after 0ms in state CONNECT, 30 identical error(s) suppressed) 2022-07-07T05:09:02Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.20.0.29:9092 failed: Connection refused (after 0ms in state CONNECT, 30 identical error(s) suppressed) 2022-07-07T05:09:08Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream caused by: error sending request for url (http://web:9000/api/0/relays/register/challenge/): operation timed out

[emmatyping]

Hm, connection refused is not good, if you installed 1 day ago, can you install a newer instance? E.g. 22.6.0 which is the latest.

What do the logs for the web container look like?

[addityar]

hello @ethanhs , our sentry version is 22.7.0.dev0

the web container logs is normal, everyting is good with web, i can access on browser normaly, but in nginx container logs showing error :

172.20.0.1 - - [07/Jul/2022:21:52:34 +0000] "POST /api/21/store/ HTTP/1.1" 502 150 "-" "sentry.php.laravel/2.10.2" "54.169.xxx.xxx" 172.20.0.1 - - [07/Jul/2022:21:52:34 +0000] "POST /api/21/store/ HTTP/1.1" 502 150 "-" "sentry.php.laravel/2.10.2" "54.169.xxx.xxx" 2022/07/07 21:52:36 [error] 21#21: *68310 connect() failed (111: Connection refused) while connecting to upstream, client: 172.20.0.1, server: , request: "POST /api/21/store/ HTTP/1.1", upstream: "http://172.20.0.22:3000/api/21/store/", host: "mydomain.com" 172.20.0.1 - - [07/Jul/2022:21:52:36 +0000] "POST /api/21/store/ HTTP/1.1" 502 150 "-" "sentry.php.laravel/2.10.2" "54.169.xxx.xxx" 172.20.0.1 - - [07/Jul/2022:21:52:36 +0000] "POST /api/21/store/ HTTP/1.1" 502 150 "-" "sentry.php.laravel/2.10.2" "54.169.xxx.xxx"

[emmatyping]

the web container logs is normal

Hmm that is odd. I would expect some error due to the refused connections in the web container. The log you posted earlier from relay is trying to make the requests against the web container. And getting connection refused.

I also cannot reproduce this, so I think it is likely an issue with your docker setup. I would recommend docker exec'ing into the containers and try to run some tests to see where the problem lies, and maybe look at logs in the nginx container.

I'll leave this issue open for now in case someone else has ideas on what could be the issue.

[itrack]

I have the same problem on 22.6.0

This is the error received in internal: SnubaError: Command # 2 (ZREMRANGEBYSCORE project_exclude_groups:errors:10 -inf 1657717905.1763825) of pipeline caused error: MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk. Commands that may modify the data set are disabled, because this instance is configured to report errors during writes if RDB snapshotting fails (stop-writes-on-bgsave-error option). Please check the Redis logs for details about the RDB error.

[emmatyping]

@itrack Are you also seeing errors similar to @addityar ? Can you please share your redis logs, it sounds like those may have more context about why it is unable to persist things to disk.

[github-actions[bot]]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀