Support multiple resource_server_id on a single RabbitMQ cluster

[MarcialRosales]

Is your feature request related to a problem? Please describe.

This is a feature requested to support the following scenarios:

• Users connect to RabbitMQ with tokens issued by more than one Authorization Server using the same audience value. Currently, this is supported provided you manually configure the corresponding signing keys used by each Authorization Server. However, very seldomly users wants to do that and instead prefer to configure the jwks_url so that RabbitMQ downloads the signing key directly from the url. Today, it is not possible to configure more than one url.
• Users connect to RabbitMQ with tokens issued by more than one Authorization Server where server uses a different audience value. Currently, RabbitMQ only supports one single resource_server_id value which means the audience has to match that single value.
• Same as the previous scenario except that in multi-tenant Authorization servers, each tenant behaves like a separate Authorization server. Furthermore, each server refers to RabbitMQ with a different audience value. The difference with regards the previous scenario is that in this one the provider_url is the same. The only thing that changes is the resource_id value.

So that you get an idea of what this feature provides, users may be authenticated against Keycloak, UAA and Oauth0 on the same RabbitMQ cluster.

Describe the solution you'd like

To keep backward compatibility, the solution proposes new settings while keeping the existing ones. The solution affects two plugins: rabbitmq-auth-backend-oauth2 and rabbitmq-management.

Example configuration of rabbitmq-auth-backend-oauth2 plugin:

# Sets common settings for all resource server(s)
auth_oauth2.scope_prefix = api://
auth_oauth2.additional_scopes_key = role
auth_oauth2.preferred_username_claims = username

# Configure RabbitMQ as an OAuth2 resource with the id "rabbitmq-for-operations" and with its jwks_uri
auth_oauth2.resource_servers.1.id = rabbitmq-for-operations
auth_oauth2.resource_servers.1.verify_aud = false
auth_oauth2.resource_servers.1.jwks_uri = https://some-uri.operations 

auth_oauth2.resource_servers.2.id = rabbitmq-for-business
auth_oauth2.resource_servers.2.jwks_uri = https://some-uri.business

If a resource_server like rabbitmq-for-business does not configure a setting like scope_prefix, it inherits from the auth_oauth2.scope_prefix setting or else from the default scope_prefix value.

If the configuration presents several auth_oauth2.resource_servers.<> ids in addition to the auth_oauth2.resource_server_id one, RabbitMQ ignores the latter.

Example configuration of rabbitmq-management plugin:

management.oauth_enabled = true
management.oauth_scopes = ....

management.resource_servers.1.id = rabbitmq-for-operations
management.resource_servers.1.label = Operations
management.resource_servers.1.client_id = client-ops
management.resource_servers.1.scopes = ....
management.resource_servers.1.provider_url = https://....

management.resource_servers.2.id = rabbitmq-for-custoners
management.resource_servers.2.client_id = client-cust
management.resource_servers.2.provider_url = https://....
management.resource_servers.2.initiated_logon_type = idp_initiated

With the above configuration, RabbitMQ users who come to management ui are presented with the typical SSO page with a drop-down with the label "Choose OAuth resource" and with the following options:

• Operations
• rabbitmq-for-customers

The list is built using the label attribute if present, else it uses the id attribute or else the index itself, e.g. 1 or 2 in the above example.

When they click on the button, they are redirected to the authorization server configured for the resource using the corresponding client_id and provider_url.

What happens with the existing oauth_ settings besides oauth_enabled ? With the above configuration, the existing oauth_ settings like oauth_scopes are used as default values. For instance, the rabbitmq-for-custoners resource does not specify scopes field. The default value is taken from management.oauth_scopes.

Is it possible to support basic auth and oauth2 authentication in the management ui? Today it is not possible. With this solution it would be possible as long as management.disable_basic_auth is false. One of the options presented in the drop-down box is:

• Basic authentication

[robcoward]

👍 My use-case for this is to allow rabbitmq clients to connect over Web MQTT protocol, passing a JWT token issued from a Keycloak IDP, while logins to the Management UI can be authenticated from a JWT token passed from Teleport (see https://goteleport.com/docs/application-access/jwt/introduction/)

[michaelklishin]

The use case we target is switching identity providers without shutting down all of your applications across all the clusters at once. For customers and OSS users with thousands of them that would basically mean halting operations for a period of time.

[MarcialRosales]

@robcoward Your use case is addressed by this feature. Will you volunteer to contribute to RabbitMQ Oauth2 tutorial repo by adding "Teleport" as a use-case? We currently have support for several Idps, but Teleport is not covered yet. And google is coming soon.

If you add the Teleport use-case, once this feature is complete, I can test it against it.

[robcoward]

@robcoward Your use case is addressed by this feature. Will you volunteer to contribute to RabbitMQ Oauth2 tutorial repo by adding "Teleport" as a use-case? We currently have support for several Idps, but Teleport is not covered yet. And google is coming soon.

If you add the Teleport use-case, once this feature is complete, I can test it against it.

Happy to. I'm not 100% sure that I'll be able to get it working yet. While Teleport exposes a .well-known/jwks.json url for verifying the JWT token it can send in the headers, it's not a full OpenID Connect IDP. The latest release of teleport has just added SAML-based IDP functionality, so I'm hoping that the jwks.json url is sufficient to be able to configure idp initated login to the management ui (there is no {oauth_provider_url, "http://localhost:8080"} url to set).

[MarcialRosales]

Thanks @robcoward . Can you paste here the content of .well-known/jwks.json?

Provided Teleport is at least Oauth2 compliant and supports authorization code with PKCE, there is an alternative although I have not implemented it yet. The idea is to configure RabbitMq management plugin with the various urls required by Oauth2 such as jwks_uri, authorize, logout. Thus, rather than configuring oauth_provider_url you would have to provide all the urls that RabbitMQ discovers via the former url.

But as I said, this is not implemented yet.

[robcoward]

Teleport acts as a reverse proxy that can pass the JWT header through to the backend rabbitmq url (I guess not too dis-similar to Oauth2Proxy in that respect) and provides the keys via the jwks.json to validate the token. It uses a wildcard dns system such that any requests to rabbitmq.teleport.example.com are initially authenticated by teleport if not already logged in, before passing the request to rabbitmq, so there isnt really a login url / oauth provider url to configure in rabbitmq other than itself. The contents of the JWT token are quite limited though so I expect I'll need to rely on setting extra_scopes_source to the roles claim, and then defining scope_aliases to map the teleport defined roles to the required rabbitmq permissions. This is what is in the jwks.json file:

{"keys":[{"kty":"RSA","alg":"RS256","n":"1ndZZQtugrL4lE2TXSRsMjVV7uS_2EKF7jVBv1vesOegS97-OCZaODTAwNFJe6s7Qintudah6o8Q9aDVDFcNLD1G84nqYxh5Ueeug3pxl_ogrYGMRj0uK67prWEXyCgzvnouBwwWfQVT0ha-yiCAO5oIudDfgOkJE4bLUTETzXQMDjsMbxJrP585sQb-5P0I1svDfp_gVW7Zj5_dgYMoEQyOxCL6OqeyDJCOlmybuVyGSjrZ60F0dfpXzFWRS2oWLv5GtMGsKElf2MzTHW0Vvqk7fvGEB6s5heF-L2z6kjrxWoNee5v92HEP8rZyHuQpdUQttcMA3WLSfcl4wJ9izQ","e":"AQAB","use":"sig","kid":""}]}

[MarcialRosales]

If users always come to RabbitMQ Management ui with a token because all requests go thru Teleport and It is Teleport who takes care of authenticating the user then this is pretty a much a Idp-initiated login scenario. As you well pointed out, same as OAuth2Proxy use-case.

This means, the oauth_provider_url is really any url where to send users when their token expires. You do not need an oauth_client_id either. Just configure {oauth_initiated_logon_type, idp_initiated},.

However, you need to configure the jwks_url in the rabbitmq_auth_backend_oauth2 plugin so that RabbitMq can validate the token. And of course, all the other settings to deal with scope mapping.

Of course, this means that all tokens are issued by Teleport. Once this feature is completed, you can define one oauth2 resource for Teleport and another for Keycloak each one of them with their own configuration.

[MarcialRosales]

@robcoward This feature is nearly completed and I have added a use-case to the OAuth2 tutorial repo to demonstrate it.

In your case, I am assuming that Teleport is actually issuing OAuth2 tokens, i.e. it delegates the authentication to 3rd party backends but ultimately Teleport is who issues the Oauth2 tokens. This means that you have two Authorization servers: Teleport and Keycloak.

This is how I would proceed :

• Configure RabbitMQ with two resources, one for keycloak with its jwks_url and another for teleport, also with its id and jwks_url.

auth_oauth2.resource_servers.1.id = rabbit_keycloak
auth_oauth2.resource_servers.1.jwks_url = https://keycloak.rabbitmq
auth_oauth2.resource_servers.2.id = rabbit_teleport
auth_oauth2.resource_servers.2.jwks_url = https://teleport.rabbitmq

• You do not need to do anything different in the management plugin, except to configure the url of Teleport in management.oauth_provider_url so that RabbitMQ can forward users to some url after their sessions expired or when users by-pass teleport and comes straight to RabbitMQ.
• Configure Teleport with the audience rabbit_teleport or with the value you chose
• Same with Keycloak Idp

Let us know if this set up works for you. The following docker image has this functionality available should you wanted to try it out: pivotalrabbitmq/rabbitmq:oauth-multi-resource-support-otp-max-bazel. It is an image built from the PR. It is not an official release.

[robcoward]

Hi @MarcialRosales - Just back from holiday and taking a look at testing out your image. I've had problems getting teleport integrated (attempted that first and will add keycloak later as there are easy examples to follow), with the jwks_url setting not working for me. I suspect it's an issue I'll raise with the teleport crew but want to run it past you first from the rabbit oauth2 perspective. I'm using the rabbit cluster operator to deploy my clusters, so use the following to setup a test deployment:

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: test
  namespace: uptimelabs
spec:
  replicas: 1
  persistence:
    storage: 1G
  override:
    statefulSet:
      spec:
        template:
          spec:
            containers:
              - name: rabbitmq
                image: pivotalrabbitmq/rabbitmq:oauth-multi-resource-support-otp-max-bazel
                volumeMounts:
                  - mountPath: /etc/rabbitmq/signing-keys
                    name: signing-keys
                    readOnly: true
            volumes:
              - name: signing-keys
                secret:
                  secretName: teleport-key
  rabbitmq:
    additionalPlugins:
      - rabbitmq_auth_backend_oauth2
    additionalConfig: |
      log.console = true
      log.console.level = debug
      auth_backends.1 = internal
      auth_backends.2.authn = rabbit_auth_backend_oauth2
      auth_backends.2.authz = internal
      management.oauth_enabled = true
      management.oauth_provider_url = https://teleport.devops-consultants.net
      management.oauth_initiated_logon_type = idp_initiated
      auth_oauth2.resource_servers.1.id = http://test.uptimelabs.svc.cluster.local:15672
      auth_oauth2.resource_servers.1.additional_scopes_key = roles
      # auth_oauth2.resource_servers.1.jwks_url = https://teleport.devops-consultants.net/.well-known/jwks.json
      auth_oauth2.default_key = "teleport"
      auth_oauth2.signing_keys.teleport = /etc/rabbitmq/signing-keys/teleport.pem
    # advancedConfig: |
    #   [
    #     {rabbitmq_auth_backend_oauth2, [
    #       {resource_server_id, <<"http://test.uptimelabs.svc.cluster.local:15672">>},
    #       {extra_scopes_source, <<"roles">>},
    #       {scope_aliases, #{
    #         <<"rabbitadmin">> => [<<"rabbitmq.tag:administrator">>]
    #       }},
    #     ]}
    #   ].

Having gone through the debug logging and the codebase, it never tries to retrieve the keys from the specified jwks_url. I suspect that it is down to the JWT token issued by teleport not including a kid attribute, and while I've specified a default_key it either still fails to retrieve the key from the url, or it attempts to, but the contents of https://teleport.devops-consultants.net/.well-known/jwks.json set a blank kid: '' which then doesnt match the default_key. If I raise an issue with the teleport team to set a value on kid, do you think this would get the jwks_url functionality working properly ?

Anyway, having converted the jwks to a pem file, I have managed to get the rabbit management code authorizing the JWT token passed from teleport, but then I face the challenge of the administrator tag. My plan was to use additional_scopes_key to pickup the roles claim in the JWT token as I dont have any control over what the JWT token contains, but using scope_aliases does not appear possible unless using the advancedConfig syntax, as as soon as I enable that, the rabbit container ends up in a crashloop cycle. I've tried a number of variations of the above syntax but none have worked - can you see anything obvious that I've missed ?

Still further, I realised that I still need to include the internal auth backend as the primary, otherwise the topology operator is unable to connect and manage queues/users etc, but then found an example setting authn & authz to different backends, so for now in addition to teleport passing a JWT for user authentication, I also have the topology operator also setting up an internal user with the administrator tag, allowing me to login to the admin UI....

Once I've added keycloak into the mix and tested a web mqtt client connection passing the JWT in the login credentials, I'll give you a further update. When I have a better teleport config sorted out, I'll contribute a use-case writeup too, as promised previously.

[michaelklishin]

IIRC this has shipped in 3.13.0 because it would be too substantial a change to consider for a patch release => set the milestone accordingly.