self-hosted
self-hosted copied to clipboard
getsentry
Auth using SAML2 with Azure AD broken after upgrade
Self-Hosted Version
24.1.1
CPU Architecture
x86_64
Docker Version
24.0.1
Docker Compose Version
v2.24.3
Steps to Reproduce
After the upgrade (see https://github.com/getsentry/self-hosted/issues/2742) I tried the SSO and failed. I removed the configuration
docker-compose run --rm web shell
...
(InteractiveConsole)
>>> from sentry.models.authprovider import AuthProvider
>>> AuthProvider.objects.all()
<BaseQuerySet [<AuthProvider at 0x7f89e3f7ffa0: id=5, organization_id=1, provider='saml2'>]>
>>> AuthProvider.objects.all().delete()
and then recreated it using the Web interface with the Auth link and SAML2 as described here https://docs.sentry.io/product/accounts/sso/azure-sso/ .
I tested the configuration in Chrome from the Microsoft Entra Admin side - successfully.
But the sign in fails with Nginx reporting a 502 bad gateway error.
Below the log from the web container.
No idea how to continue from here.
Expected Result
Single Sign On using SAML2 provider should work like it used to...
Actual Result
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/sentry/../sentry_sdk/integrations/django/__init__.py", line 165, in sentry_patched_wsgi_handler
return SentryWsgiMiddleware(bound_old_app, use_x_forwarded_for)(
File "/usr/local/lib/python3.10/site-packages/sentry/../sentry_sdk/integrations/wsgi.py", line 115, in __call__
reraise(*_capture_exception(hub))
File "/usr/local/lib/python3.10/site-packages/sentry/../sentry_sdk/_compat.py", line 115, in reraise
raise value
File "/usr/local/lib/python3.10/site-packages/sentry/../sentry_sdk/integrations/wsgi.py", line 108, in __call__
rv = self.app(
File "/usr/local/lib/python3.10/site-packages/django/core/handlers/wsgi.py", line 124, in __call__
response = self.get_response(request)
File "/usr/local/lib/python3.10/site-packages/sentry/../sentry_sdk/integrations/django/__init__.py", line 460, in sentry_patched_get_response
rv = old_get_response(self, request)
File "/usr/local/lib/python3.10/site-packages/django/core/handlers/base.py", line 140, in get_response
response = self._middleware_chain(request)
File "/usr/local/lib/python3.10/site-packages/django/core/handlers/exception.py", line 57, in inner
response = response_for_exception(request, exc)
File "/usr/local/lib/python3.10/site-packages/django/core/handlers/exception.py", line 140, in response_for_exception
response = handle_uncaught_exception(
File "/usr/local/lib/python3.10/site-packages/django/core/handlers/exception.py", line 185, in handle_uncaught_exception
return callback(request)
File "/usr/local/lib/python3.10/site-packages/django/views/generic/base.py", line 104, in view
return self.dispatch(request, *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/sentry/web/frontend/error_500.py", line 37, in dispatch
return render_to_response("sentry/500.html", status=500, context=context, request=request)
File "/usr/local/lib/python3.10/site-packages/sentry/web/helpers.py", line 42, in render_to_response
response = HttpResponse(render_to_string(template, context, request))
File "/usr/local/lib/python3.10/site-packages/sentry/web/helpers.py", line 29, in render_to_string
rendered = loader.render_to_string(template, context=context, request=request)
File "/usr/local/lib/python3.10/site-packages/django/template/loader.py", line 62, in render_to_string
return template.render(context, request)
File "/usr/local/lib/python3.10/site-packages/django/template/backends/django.py", line 61, in render
return self.template.render(context)
File "/usr/local/lib/python3.10/site-packages/django/template/base.py", line 175, in render
return self._render(context)
File "/usr/local/lib/python3.10/site-packages/django/template/base.py", line 167, in _render
return self.nodelist.render(context)
File "/usr/local/lib/python3.10/site-packages/django/template/base.py", line 1005, in render
return SafeString("".join([node.render_annotated(context) for node in self]))
File "/usr/local/lib/python3.10/site-packages/django/template/base.py", line 1005, in <listcomp>
return SafeString("".join([node.render_annotated(context) for node in self]))
File "/usr/local/lib/python3.10/site-packages/django/template/base.py", line 966, in render_annotated
return self.render(context)
File "/usr/local/lib/python3.10/site-packages/django/template/loader_tags.py", line 157, in render
return compiled_parent._render(context)
File "/usr/local/lib/python3.10/site-packages/django/template/base.py", line 167, in _render
return self.nodelist.render(context)
File "/usr/local/lib/python3.10/site-packages/django/template/base.py", line 1005, in render
return SafeString("".join([node.render_annotated(context) for node in self]))
File "/usr/local/lib/python3.10/site-packages/django/template/base.py", line 1005, in <listcomp>
return SafeString("".join([node.render_annotated(context) for node in self]))
File "/usr/local/lib/python3.10/site-packages/django/template/base.py", line 966, in render_annotated
return self.render(context)
File "/usr/local/lib/python3.10/site-packages/django/template/loader_tags.py", line 157, in render
return compiled_parent._render(context)
File "/usr/local/lib/python3.10/site-packages/django/template/base.py", line 167, in _render
return self.nodelist.render(context)
File "/usr/local/lib/python3.10/site-packages/django/template/base.py", line 1005, in render
return SafeString("".join([node.render_annotated(context) for node in self]))
File "/usr/local/lib/python3.10/site-packages/django/template/base.py", line 1005, in <listcomp>
return SafeString("".join([node.render_annotated(context) for node in self]))
File "/usr/local/lib/python3.10/site-packages/django/template/base.py", line 966, in render_annotated
return self.render(context)
File "/usr/local/lib/python3.10/site-packages/django/template/loader_tags.py", line 63, in render
result = block.nodelist.render(context)
File "/usr/local/lib/python3.10/site-packages/django/template/base.py", line 1005, in render
return SafeString("".join([node.render_annotated(context) for node in self]))
File "/usr/local/lib/python3.10/site-packages/django/template/base.py", line 1005, in <listcomp>
return SafeString("".join([node.render_annotated(context) for node in self]))
File "/usr/local/lib/python3.10/site-packages/django/template/base.py", line 966, in render_annotated
return self.render(context)
File "/usr/local/lib/python3.10/site-packages/sentry/templatetags/sentry_assets.py", line 90, in render
content = self.nodelist.render(context).strip()
File "/usr/local/lib/python3.10/site-packages/django/template/base.py", line 1005, in render
return SafeString("".join([node.render_annotated(context) for node in self]))
File "/usr/local/lib/python3.10/site-packages/django/template/base.py", line 1005, in <listcomp>
return SafeString("".join([node.render_annotated(context) for node in self]))
File "/usr/local/lib/python3.10/site-packages/django/template/base.py", line 966, in render_annotated
return self.render(context)
File "/usr/local/lib/python3.10/site-packages/django/template/library.py", line 237, in render
output = self.func(*resolved_args, **resolved_kwargs)
File "/usr/local/lib/python3.10/site-packages/sentry/templatetags/sentry_react.py", line 11, in get_react_config
context = get_client_config(context.get("request", None), context.get("org_context"))
File "/usr/local/lib/python3.10/site-packages/sentry/web/client_config.py", line 384, in get_client_config
return config.get_context()
File "/usr/local/lib/python3.10/site-packages/sentry/web/client_config.py", line 371, in get_context
"user": self.user_details,
File "/usr/local/lib/python3.10/functools.py", line 981, in __get__
val = self.func(instance)
File "/usr/local/lib/python3.10/site-packages/sentry/web/client_config.py", line 290, in user_details
user_details["isSuperuser"] = self.user.is_superuser
TypeError: 'NoneType' object does not support item assignment
Event ID
No response
I suspect that there is some corrupt data in postgres. Can you make sure the superuser flag is set in the auth_user table?
Hi @hubertdeng123 ,
just checked, yes, I have 4 users which have is_superuser set to true.
root@sentry01(dc1.prd):~# docker exec -it sentry-self-hosted-postgres-1 psql -U postgres
psql (14.5 (Debian 14.5-2.pgdg110+2))
Type "help" for help.
postgres=# select count(*) from auth_user where is_superuser = 't';
count
-------
4
(1 row)
postgres=#
When I log on with SAML disabled as a superuser, I get this:
Clicking "If you need to make changes to your user account, please click here." takes me to the organization.
There creating a new "local" user sends out the eMail, when clicking on the link it gives
Is sentry_organizationmapping supposed to be empty?
postgres=# select * from sentry_organizationmapping;
id | organization_id | slug | name | date_created | customer_id | verified | idempotency_key | region_name | status | require_2fa | early_adopter | allow_joinleave | enhanced_privacy | disable_shared_issues | disable_new_visibility_features | require_email_verification | codecov
_access
----+-----------------+------+------+--------------+-------------+----------+-----------------+-------------+--------+-------------+---------------+-----------------+------------------+-----------------------+---------------------------------+----------------------------+--------
--------
(0 rows)
The organization invite link is no longer valid.
Did you make sure to logout or clear cookies before clicking on the link?
The sentry_organizationmapping is supposed to map org slugs to a specific org and region, which should not be applicable here.
After creating a new entry in the empty sentry_organizationmapping PostGres table we were able to login again and got routed correctly to the organization again and were successfully able to activate SAML and the integrations.
Could you search your web container logs for the following regex:
service.py", line \d+, in serialize_many
I'm curious if there is an async function that is causing trouble here.
After creating a new entry in the empty sentry_organizationmapping PostGres table we were able to login again and got routed correctly to the organization again and were successfully able to activate SAML and the integrations.
Looks like I was wrong, great find here!
service.py", line \d+, in serialize_many
Tried that, but nothing in the logs...
root@sentry01(dc1.prd):/var/lib/docker/containers# grep -R "service.py\", line \d+, in serialize_many" *
root@sentry01(dc1.prd):/var/lib/docker/containers#
After creating a new entry in the empty sentry_organizationmapping PostGres table we were able to login again and got routed correctly to the organization again and were successfully able to activate SAML and the integrations.
Looks like I was wrong, great find here!
@hubertdeng123, I believe more people will come across that missing entry - is there any way to find out how it got deleted in the first place?
I wonder if we can just put something in the next release to rebuild the organization_mapping? Seems like we could just wipe the table and rebuild it based on existing entries in the orgnization table directly.
After creating a new entry in the empty sentry_organizationmapping PostGres table we were able to login again and got routed correctly to the organization again and were successfully able to activate SAML and the integrations.
@stumbaumr Can you give a bit more detail on how you add those data ? We are affected by the same issue when trying to integration Authentik SAML in our current Sentry setup
Start a psql session in your postgres container and insert an entry into that table using a INSERT statement...
root@sentry01(dc1.prd):~# docker exec -it sentry-self-hosted-postgres-1 psql -U postgres
psql (14.5 (Debian 14.5-2.pgdg110+2))
Type "help" for help.
postgres=# select * from sentry_organizationmapping;
id | organization_id | slug | name | date_created | customer_id | verified | idempotency_key | region_name | status | require_2fa | early_adopter | allow_joinleave | enhanced_privacy | disable_shared_issues | disable_new_visibility_features | require_email_veri
fication | codecov_access
----+-----------------+--------+-----------+------------------------+-------------+----------+-----------------+--------------+--------+-------------+---------------+-----------------+------------------+-----------------------+---------------------------------+-------------------
---------+----------------
3 | 1 | XXX | XXX | 2024-01-01 00:00:00+00 | XXX | t | XXX | --monolith-- | 0 | f | f | t | f | f | f | f
| f
(1 row)
postgres=#
@azaslavsky The more users upgrade the more support is required - maybe really best to just provide a delete and recreate procedure...
This happend to our self-hosted instance aswell. There was a lot of repeating stacktrace.
web-1 | During handling of the above exception, another exception occurred:
web-1 |
web-1 | Traceback (most recent call last):
web-1 | File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
web-1 | response = get_response(request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/../sentry_sdk/integrations/django/middleware.py", line 175, in __call__
web-1 | return f(*args, **kwargs)
web-1 | ^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/deprecation.py", line 134, in __call__
web-1 | response = response or self.get_response(request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 57, in inner
web-1 | response = response_for_exception(request, exc)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 140, in response_for_exception
web-1 | response = handle_uncaught_exception(
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 185, in handle_uncaught_exception
web-1 | return callback(request)
web-1 | ^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 104, in view
web-1 | return self.dispatch(request, *args, **kwargs)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/web/frontend/error_500.py", line 37, in dispatch
web-1 | return render_to_response("sentry/500.html", status=500, context=context, request=request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/web/helpers.py", line 43, in render_to_response
web-1 | response = HttpResponse(render_to_string(template, context, request))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/web/helpers.py", line 30, in render_to_string
web-1 | rendered = loader.render_to_string(template, context=context, request=request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/loader.py", line 62, in render_to_string
web-1 | return template.render(context, request)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/backends/django.py", line 61, in render
web-1 | return self.template.render(context)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 171, in render
web-1 | return self._render(context)
web-1 | ^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 163, in _render
web-1 | return self.nodelist.render(context)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 1000, in render
web-1 | return SafeString("".join([node.render_annotated(context) for node in self]))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 1000, in <listcomp>
web-1 | return SafeString("".join([node.render_annotated(context) for node in self]))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 961, in render_annotated
web-1 | return self.render(context)
web-1 | ^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/loader_tags.py", line 159, in render
web-1 | return compiled_parent._render(context)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 163, in _render
web-1 | return self.nodelist.render(context)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 1000, in render
web-1 | return SafeString("".join([node.render_annotated(context) for node in self]))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 1000, in <listcomp>
web-1 | return SafeString("".join([node.render_annotated(context) for node in self]))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 961, in render_annotated
web-1 | return self.render(context)
web-1 | ^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/loader_tags.py", line 159, in render
web-1 | return compiled_parent._render(context)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 163, in _render
web-1 | return self.nodelist.render(context)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 1000, in render
web-1 | return SafeString("".join([node.render_annotated(context) for node in self]))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 1000, in <listcomp>
web-1 | return SafeString("".join([node.render_annotated(context) for node in self]))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 961, in render_annotated
web-1 | return self.render(context)
web-1 | ^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/loader_tags.py", line 65, in render
web-1 | result = block.nodelist.render(context)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 1000, in render
web-1 | return SafeString("".join([node.render_annotated(context) for node in self]))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 1000, in <listcomp>
web-1 | return SafeString("".join([node.render_annotated(context) for node in self]))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 961, in render_annotated
web-1 | return self.render(context)
web-1 | ^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/templatetags/sentry_assets.py", line 90, in render
web-1 | content = self.nodelist.render(context).strip()
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 1000, in render
web-1 | return SafeString("".join([node.render_annotated(context) for node in self]))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 1000, in <listcomp>
web-1 | return SafeString("".join([node.render_annotated(context) for node in self]))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 961, in render_annotated
web-1 | return self.render(context)
web-1 | ^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/django/template/library.py", line 237, in render
web-1 | output = self.func(*resolved_args, **resolved_kwargs)
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/templatetags/sentry_react.py", line 11, in get_react_config
web-1 | context = get_client_config(context.get("request", None), context.get("org_context"))
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/web/client_config.py", line 412, in get_client_config
web-1 | return config.get_context()
web-1 | ^^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/web/client_config.py", line 397, in get_context
web-1 | "user": self.user_details,
web-1 | ^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__
web-1 | val = self.func(instance)
web-1 | ^^^^^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.11/site-packages/sentry/web/client_config.py", line 316, in user_details
web-1 | user_details["isSuperuser"] = self.user.is_superuser
web-1 | ~~~~~~~~~~~~^^^^^^^^^^^^^^^
web-1 | TypeError: 'NoneType' object does not support item assignment
Just added this stacktrace because it doesn't look all the same from the issue creator. (i could be wrong tho)
We're still thinking about what the best approach here is. Automatically adjusting users organizationmapping tables requires making a lot of assumptions that might hurt more users than it helps.
Maybe just a proper sample INSERT-SELECT statement in this thread so people know what to tweak. This is not enduser level stuff we are talking about here...
I wonder if your issue popped up after hybrid cloud changes were implemented. At what point did you realize SAML2 was broken? Was it working for your 23.6.2 upgrade?
This flow may very well be broken for everyone so it would be nice to have some sort of long term fix, even though an INSERT-SELECT may do the job for now.
+1 I'm trying to upgrade sentry from 23.3.1 to 24.2.0 and have the same issue. In 23.3.1 the table sentry_organizationmapping is empty.
Same issue here. A way to populate the sentry_organizationmapping table would be appreciated.
@stephanepechard look at the message from @stumbaumr above. https://github.com/getsentry/self-hosted/issues/2750#issuecomment-1954094951
Everybody with self hosting and external authentication will be impacted after the upgrade. As long as Devs don't know at which version step the problem occurs they can't provide a fix...
After redoing all the step from a snapshot, it's working without issues. Probably I missed a step or some migration failed. but now I don't have any issues with Okta.
22.10 -> 23.6.2(hard stop) -> 24.2