Symbolicator not working anymore

[farfromrefug]

Self-Hosted Version

24.11.1

CPU Architecture

x86_64

Docker Version

24.0.2

Docker Compose Version

2.20.1

Machine Specification

• [x] My system meets the minimum system requirements of Sentry

Steps to Reproduce

Lately i realized that my sentry errors were not symbolicated anymore. I did not change anything with my setup. I might have run docker-compose down /up but that's it. Looking at the docker logs i saw that in my symbolicator container:

DEBUG symbolicate_js:fetch_minified_and_sourcemap:get_file: symbolicator_js::api_lookup: Fetching list of Sentry JS artifacts from https://myhost/api/0/projects/akylas/documentscanner/artifact-lookup/?release=com.akylas.documentscanner%401.14.5%2B121&dist=121.android.playstore&debug_id=d242e012-4b50-49b6-a5d9-dcd689446ad6 debug_id=Some(DebugId { uuid: \"d242e012-4b50-49b6-a5d9-dcd689446ad6\", appendix: 0 }) abs_path=file:///data/data/com.akylas.documentscanner/files/app/vendor.js key=SourceMap { abs_path: None, debug_id: Some(DebugId { uuid: \"d242e012-4b50-49b6-a5d9-dcd689446ad6\", appendix: 0 }) }
WARN fetch_sentry_json: symbolicator_service::download::sentry: Sentry API returned status code 401 Unauthorized

If i open the link to the artifact in my browser it works i can get the file. Does anyone have any idea why the container would have a 401 error on that file?

Expected Result

symbolicator working

Actual Result

DEBUG symbolicate_js:fetch_minified_and_sourcemap:get_file: symbolicator_js::api_lookup: Fetching list of Sentry JS artifacts from https://myhost/api/0/projects/akylas/documentscanner/artifact-lookup/?release=com.akylas.documentscanner%401.14.5%2B121&dist=121.android.playstore&debug_id=d242e012-4b50-49b6-a5d9-dcd689446ad6 debug_id=Some(DebugId { uuid: \"d242e012-4b50-49b6-a5d9-dcd689446ad6\", appendix: 0 }) abs_path=file:///data/data/com.akylas.documentscanner/files/app/vendor.js key=SourceMap { abs_path: None, debug_id: Some(DebugId { uuid: \"d242e012-4b50-49b6-a5d9-dcd689446ad6\", appendix: 0 }) }
WARN fetch_sentry_json: symbolicator_service::download::sentry: Sentry API returned status code 401 Unauthorized

Event ID

No response

[aldy505]

@loewenheim Did something change with symbolicator? Do you know how to debug this further?

[loewenheim]

Hi, apologies, I was on vacation. Which version of Symbolicator is this?

[farfromrefug]

@loewenheim i am on 24.11.1

[loewenheim]

Did you change anything about the system.internal-url-prefix Sentry setting recently? In my experience that is a common source of Symbolicator/Sentry communication problems.

[farfromrefug]

@loewenheim no i did not. But is that normal for it to use external url. Shouldnt it use dockrr link url? Right now it means it goes over internet when everything should be local and even more only inside docker network

[gstorme]

Also seeing this on 25.7.0 Sentry is behind a haproxy loadbalancer.

Config:

system.internal-url-prefix: 'https://sentry.masked-ops.net'
system.url-prefix: 'https://sentry.masked-ops.net'
symbolicator.enabled: true
symbolicator.options:
  url: "http://symbolicator:3021"

Sentry logs:

2025-10-20T13:14:26.420358127Z  WARN symbolicator_service::download::sentry: Sentry API returned status code 401 Unauthorized
2025-10-20T13:14:26.420508264Z ERROR symbolicator_js::lookup: Failed to query Sentry for files error=download failed: 401 Unauthorized lookup_url=https://sentry.masked-ops.net/api/0/projects/masked/whitelabel-controlpanel-js/artifact-lookup/

Haproxy logs:

Oct 20 15:11:03 tb-lb001 haproxy[804916]: x.x.174.43:48576 [20/Oct/2025:15:11:03.735] f_frontend001~ b_sentry/tb-sentry001 0/0/0/28/28 401 1033 - - ---- 86/83/0/0/0 0/0 "GET /api/0/projects/masked/whitelabel-controlpanel-js/artifact-lookup/?release=0.1.0&url=%2Fbuild%2Fassets%2Fapp-DRxLoBZd&debug_id=87750f61-c8f0-4473-9f70-a2e8b9e44254 HTTP/1.1"
Oct 20 15:11:03 tb-lb001 haproxy[804916]: x.x.174.43:48576 [20/Oct/2025:15:11:03.785] f_frontend001~ b_sentry/tb-sentry001 0/0/0/29/29 401 1033 - - ---- 85/82/0/0/0 0/0 "GET /api/0/projects/masked/whitelabel-controlpanel-js/artifact-lookup/?release=0.1.0&url=%2Fbuild%2Fassets%2Fapp-DRxLoBZd&debug_id=87750f61-c8f0-4473-9f70-a2e8b9e44254 HTTP/1.1"

The IP address x.x.174.43 is of the sentry server.

[loewenheim]

@gstorme Did this also recently stop working for you?

[gstorme]

I'm seeing this error since we upgraded to 25.7.0 in August. Can't say when this started exactly for us, since I don't have the logs from before 25.7.0.

[loewenheim]

That error message was only introduced in April, so it's possible that this had already been happening and wasn't caused by the update.

[gstorme]

For me, this is fixed after adding set_real_ip_from with the IP's of the loadbalancer to nginx.conf, and adding the IP of the sentry server to INTERNAL_SYSTEM_IPS in sentry.conf.py. Followed by running install.sh

[getsantry[bot]]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

[lemrouch]

Hi, do you happen to have http_proxy set? I've found client denied by server configuration: proxy:http://0.0.0.0:3021/healthcheck and client denied by server configuration: proxy:http://0.0.0.0:3000/api/relay/healthcheck/live/ in our HTTP proxy logs after upgrade to 25.11.1.

Emptying HTTP_PROXY in docker-compose.yml fixed it:

@@ -469,6 +474,7 @@ services:
     image: "$SYMBOLICATOR_IMAGE"
     environment:
       SYMBOLICATOR_STATSD_ADDR: ${STATSD_ADDR:-127.0.0.1:8125}
+      HTTP_PROXY: ""
     command: run -c /etc/symbolicator/config.yml
     volumes:
       - "sentry-symbolicator:/data"
@@ -707,6 +713,7 @@ services:
     image: "$RELAY_IMAGE"
     environment:
       RELAY_STATSD_ADDR: ${STATSD_ADDR:-127.0.0.1:8125}
+      HTTP_PROXY: ""
     volumes:
       - type: bind
         read_only: true

[lemrouch]

I would say there should be localhost in the noProxy list on installing-behind-a-proxy. BTW: nginx uses localhost in healthcheck too.