Setting Account Security and Notification Stop to work

[aamarques]

Environment

self-hosted (https://develop.sentry.dev/self-hosted/)

Steps to Reproduce

Just navigate to Settings/General Settings/Security or Settings/General Settings/Notification

Expected Result

The UI working

Actual Result

I have the same problem on 24.8.0, 24.9.0 and 24.10.0 versions

This is the image of Security:

This is the image for Notification (infinite loop)

On the web container log, I have a lot of the following lines :

web-1  | 17:44:49 [WARNING] django.request: Unauthorized: /api/0/projects/ccmn/node-chat-proxy/artifact-lookup/ (status_code=401 request=<WSGIRequest: GET '/api/0/projects/ccmn/node-chat-proxy/artifact-lookup/?release=aee97488df4f95f97ac721f939df8e0eb49fa238&url=%2Fapp'>)
web-1  | 17:44:49 [ERROR] sentry.auth.system: Trying to use `SystemToken` from non-internal IP
web-1  | 17:44:49 [ERROR] sentry.auth.system: Trying to use `SystemToken` from non-internal IP
web-1  | 17:44:49 [INFO] sentry.access.api: api.access (method='GET' view='sentry.api.endpoints.artifact_lookup.ProjectArtifactLookupEndpoint' response=401 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/0/projects/ccmn/node-chat-proxy/artifact-lookup/' caller_ip='XXX.XXX.XXX.XXX' user_agent='symbolicator/24.10.0' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.03459882736206055 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None' token_last_characters='2c2b')
web-1  | 17:44:49 [WARNING] django.request: Unauthorized: /api/0/projects/ccmn/node-chat-proxy/artifact-lookup/ (status_code=401 request=<WSGIRequest: GET '/api/0/projects/ccmn/node-chat-proxy/artifact-lookup/?release=aee97488df4f95f97ac721f939df8e0eb49fa238&url=%2Fapp'>)

Product Area

Settings - General

Link

No response

DSN

No response

Version

24.10.0

[getsantry[bot]]

Assigning to @getsentry/support for routing ⏲️

[aamarques]

Any ideas ?

[bc-sentry]

We are unable to reproduce this issue. We suspect it may be related to firewall and/or network settings. Please examine your container configuration to see if you are restricting traffic. Thank you.

[aamarques]

We are unable to reproduce this issue. We suspect it may be related to firewall and/or network settings. Please examine your container configuration to see if you are restricting traffic. Thank you.

Ok, but until I know, we haven't changed the configurations.

[bc-sentry]

I see. If you find additional info, please re-open with additional logs.

[aamarques]

Following #3353

[aamarques]

There are a lot of users with the same issue, so this can't be a firewall issue.

[aldy505]

I'm going to reopen this issue.

[hubertdeng123]

Here is the line of code that is throwing the error: https://github.com/getsentry/sentry/blob/master/src/sentry/auth/system.py#L51

I am wondering if your symbolicator service is not being recognized as an internal IP

[aamarques]

Hey @aldy505 and @hubertdeng123

The Notification and Security page seems to be another "error". But I cannot identify it. The Token error I did a workarround and seems to work (see my comment on https://github.com/getsentry/self-hosted/issues/3353#issuecomment-2503222058).

I appreciate any help on that.

[aamarques]

These are the Logs when I click on "Security" link. The INTERNAL and EXTERNAL IP are in the INTERNAL_SYSTEM_IPS

web-1                                           | 09:42:13 [INFO] sentry.access.api: api.access (method='GET' view='sentry.users.api.endpoints.user_emails.UserEmailsEndpoint' response=200 user_id='14' is_app='False' token_type='None' is_frontend_request='True' organization_id='None' auth_id='None' path='/api/0/users/me/emails/' caller_ip='<EXTERNAL_IP>' user_agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.02071666717529297 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
nginx-1                                         | <EXTERNAL_IP> - - [27/Nov/2024:09:42:13 +0000] "GET /api/0/users/me/emails/ HTTP/1.1" 200 83 "https://sentry.wl.myccn.org/settings/account/security/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "<INTERNAL_IP>"
web-1                                           | 09:42:13 [INFO] sentry.access.api: api.access (method='GET' view='sentry.users.api.endpoints.user_authenticator_index.UserAuthenticatorIndexEndpoint' response=200 user_id='14' is_app='False' token_type='None' is_frontend_request='True' organization_id='None' auth_id='None' path='/api/0/users/me/authenticators/' caller_ip='<EXTERNAL_IP>' user_agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.035387277603149414 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
nginx-1                                         | <EXTERNAL_IP> - - [27/Nov/2024:09:42:13 +0000] "GET /api/0/users/me/authenticators/ HTTP/1.1" 200 1824 "https://sentry.wl.myccn.org/settings/account/security/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "<INTERNAL_IP>"
web-1                                           | 09:42:13 [INFO] sentry.access.api: api.access (method='GET' view='sentry.web.frontend.react_page.ReactPageView' response=200 user_id='14' is_app='False' token_type='None' is_frontend_request='True' organization_id='1' auth_id='None' path='//api/0/organizations/' caller_ip='<EXTERNAL_IP>' user_agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.12106037139892578 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
nginx-1                                         | <EXTERNAL_IP> - - [27/Nov/2024:09:42:13 +0000] "GET //api/0/organizations/ HTTP/1.1" 200 8122 "https://sentry.wl.myccn.org/settings/account/security/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "<INTERNAL_IP>"
web-1                                           | 09:42:14 [INFO] sentry.access.api: api.access (method='GET' view='sentry.web.frontend.react_page.ReactPageView' response=200 user_id='14' is_app='False' token_type='None' is_frontend_request='True' organization_id='1' auth_id='None' path='//api/0/organizations/' caller_ip='<EXTERNAL_IP>' user_agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.095458984375 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
nginx-1                                         | <EXTERNAL_IP> - - [27/Nov/2024:09:42:14 +0000] "GET //api/0/organizations/ HTTP/1.1" 200 8116 "https://sentry.wl.myccn.org/settings/account/security/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "<INTERNAL_IP>"
web-1                                           | 09:42:17 [INFO] sentry.access.api: api.access (method='GET' view='sentry.web.frontend.react_page.ReactPageView' response=200 user_id='14' is_app='False' token_type='None' is_frontend_request='True' organization_id='1' auth_id='None' path='//api/0/organizations/' caller_ip='<EXTERNAL_IP>' user_agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.0957634449005127 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
nginx-1                                         | <EXTERNAL_IP> - - [27/Nov/2024:09:42:17 +0000] "GET //api/0/organizations/ HTTP/1.1" 200 8141 "https://sentry.wl.myccn.org/settings/account/security/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "<INTERNAL_IP>"
web-1                                           | 09:42:21 [INFO] sentry.access.api: api.access (method='GET' view='sentry.web.frontend.react_page.ReactPageView' response=200 user_id='14' is_app='False' token_type='None' is_frontend_request='True' organization_id='1' auth_id='None' path='//api/0/organizations/' caller_ip='<EXTERNAL_IP>' user_agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.09476971626281738 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
nginx-1                                         | <EXTERNAL_IP> - - [27/Nov/2024:09:42:21 +0000] "GET //api/0/organizations/ HTTP/1.1" 200 8106 "https://sentry.wl.myccn.org/settings/account/security/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "<INTERNAL_IP>"

Then I got :

[bc-sentry]

@aamarques I have tried to reproduce this again on our internal self-hosted instance and cannot. The logs you show in your latest message are all HTTP 200. Given the "error loading data" screen, it would seem that there still must be an HTTP 4xx occurring. Do you see that or an HTTP 5xx in the logs?

[aamarques]

@bc-sentry That's is strange. I have a test machine builder from the production data. There's no 5xx or 4xx in logs of web-1 or nginx-1 instances. Is there another place to search ? Or something related to permission or else ?

[aamarques]

I updated it to 24.11.1, and stills happening :(

[aldy505]

Chiming in. I can't reproduce this either. Can you provide some more details about:

1. Your Docker daemon configurations. Possibly on /etc/docker/daemon.json if you're running Docker in regular root mode.
2. Your underlying infrastructure networking, meaning your private IP and subnet range, does it conflict from what's being defined on the Docker IP address pool?
3. If you're using a reverse proxy in front of the entire Sentry deployment, how did you configure it?

[aamarques]

Chiming in. I can't reproduce this either. Can you provide some more details about:

1. Your Docker daemon configurations. Possibly on /etc/docker/daemon.json if you're running Docker in regular root mode. This is systemd

3. Your underlying infrastructure networking, meaning your private IP and subnet range, does it conflict from what's being defined on the Docker IP address pool? No conflict. Sentry is 172.x, and host is 10.6.x

5. If you're using a reverse proxy in front of the entire Sentry deployment, how did you configure it? No reverse Proxy.

[aamarques]

No ideas ? :(

[jjw99]

I'm seeing the same thing on 25.1.0. The security and notifications settings pages fail to load. Only 200 status codes in chrome debug -> Network tab.

[jjw99]

There seems to be something related to getting information about the organization. On the security page, accountSecurityWrapper.tsx, reports that orgRequest.isError = true.

And on the notifications page, notificationSettingsController.tsx, organizationsLoading is always true:

It might be due to the double slash //api here?

---

Edit: I removed the trailing "/" from my sentry/config.yml -> system.url-prefix value and it resolved this issue. Not sure why it is only a problem on these two endpoints. FYI - @aamarques

[antonio-marques-ccmn]

Hey @jjw99, <UPDATE> Thank you for being so helpful. I made the changes in sentry/config.yml -> system.url-prefix, removing the last slash "/" and WORKS!!!!

system.url-prefix: 'https://sentry.xxx.xxx..org'
symbolicator.enabled: true
symbolicator.options:
   url: "http://symbolicator:3021"

Thank you guy @jjw99

@hubertdeng123 @aldy505 What do you think about this analysis ?

Thanks all!