dataverse API OIDC authentication mechanism

What this PR does / why we need it: It reimplements OIDC authentication mechanism (including the bearer tokens authentication)

Which issue(s) this PR closes:

Closes OIDC/bearer token related issues

Special notes for your reviewer: This is very similar conceptually as the oauth2-proxy, as in https://github.com/IQSS/dataverse-frontend/pull/504, however, it turns out that the needed functionality is already supported in Payara: https://docs.payara.fish/enterprise/docs/Technical%20Documentation/Public%20API/OpenID%20Connect%20Support.html This PR eliminates any need for a proxy, etc. It turned out that the implementation is very simple and elegant, i.m.h.o.

Suggestions on how to test this:

it is a good idea to test the entire flow and make sure the user you are testing with does not yet exist in the dev dataverse instance, but it does exist keycloack. You can delete your docker volumes with sudo rm -rf docker-dev-volumes, if that helps. Also, make sure that you have an entry for the Kaycloak in your hosts file (e.g., /etc/hosts on unix system), as described in the docker-compose file. It should look soemthing like this:

127.0.1.1       keycloak.mydomain.com

you can run the dev env with mvn -Pct clean package docker:run
go to http://localhost:8080/ and click 'Log In', click "OpenID Connect" and then click "Log In with OpenID Connect"
log in with admin/admin

you will get the redirect to create a new user (because of the first time log in)

choose a user name and agree to the conditions, click on "Create Account" and now you are logged in:

log out and log in with OpenID connect again, to make sure that works
test the API call with going in the browser to http://localhost:8080/api/v1/users/:me

test the session is working by going (still in the browser) to http://localhost:8080/api/v1/oidc/session

copy the "session" field and try curl (replace the session-id with the copied value)

curl -v --cookie "JSESSIONID=session-id" http://localhost:8080/api/v1/users/:me | jq .

test the API/Bearer token flow by running the Python script, e.g.:

cd doc/sphinx-guides/_static/api/bearer-token-example
./run.sh

this will start a Python script that will prompt a log in in a new browser window:

log in with admin/admin
Python will get the bearer token and call http://localhost:8080/api/v1/users/:me

Does this PR introduce a user interface change? If mockups are available, please link/include them here: No

Is there a release notes update needed for this change?: Yes

Additional documentation: https://docs.payara.fish/enterprise/docs/Technical%20Documentation/Public%20API/OpenID%20Connect%20Support.html

Doc preview: https://dataverse-guide--10905.org.readthedocs.build/en/10905/installation/oidc.html

Oct 03 '24 15:10 ErykKul

coverage: 22.466% (-0.3%) from 22.76% when pulling 76359d110611213574803348ae59c4a6d5b7a4dc on authn-arch-api-oidc into 78ac64b484a4db0e1b8a596307b9350a50e56344 on develop.

Oct 03 '24 15:10 coveralls

:package: Pushed preview images as

ghcr.io/gdcc/dataverse:authn-arch-api-oidc

ghcr.io/gdcc/configbaker:authn-arch-api-oidc

:ship: See on GHCR. Use by referencing with full name as printed above, mind the registry name.

Oct 03 '24 15:10 github-actions[bot]

:package: Pushed preview images as

ghcr.io/gdcc/dataverse:authn-arch-api-oidc

ghcr.io/gdcc/configbaker:authn-arch-api-oidc

:ship: See on GHCR. Use by referencing with full name as printed above, mind the registry name.

Oct 03 '24 15:10 github-actions[bot]

:package: Pushed preview images as

ghcr.io/gdcc/dataverse:authn-arch-api-oidc

ghcr.io/gdcc/configbaker:authn-arch-api-oidc

:ship: See on GHCR. Use by referencing with full name as printed above, mind the registry name.

Oct 03 '24 18:10 github-actions[bot]

I forgot to mention this: the log-in is a one time thing. The normal flow is to go to http://localhost:8080/api/v1/callback/session and only if you are not authenticated (I implemented returning better error message now), you go to http://localhost:8080/oidc/login.

@pdurbin thanks for the docker run command, I will try testing it with that i.s.o. my own dev environment.

Oct 03 '24 18:10 ErykKul

:package: Pushed preview images as

ghcr.io/gdcc/dataverse:authn-arch-api-oidc

ghcr.io/gdcc/configbaker:authn-arch-api-oidc

:ship: See on GHCR. Use by referencing with full name as printed above, mind the registry name.

Oct 03 '24 18:10 github-actions[bot]

:package: Pushed preview images as

ghcr.io/gdcc/dataverse:authn-arch-api-oidc

ghcr.io/gdcc/configbaker:authn-arch-api-oidc

:ship: See on GHCR. Use by referencing with full name as printed above, mind the registry name.

Oct 03 '24 19:10 github-actions[bot]

:package: Pushed preview images as

ghcr.io/gdcc/dataverse:authn-arch-api-oidc

ghcr.io/gdcc/configbaker:authn-arch-api-oidc

:ship: See on GHCR. Use by referencing with full name as printed above, mind the registry name.

Oct 03 '24 19:10 github-actions[bot]

:package: Pushed preview images as

ghcr.io/gdcc/dataverse:authn-arch-api-oidc

ghcr.io/gdcc/configbaker:authn-arch-api-oidc

:ship: See on GHCR. Use by referencing with full name as printed above, mind the registry name.

Oct 03 '24 19:10 github-actions[bot]

:package: Pushed preview images as

ghcr.io/gdcc/dataverse:authn-arch-api-oidc

ghcr.io/gdcc/configbaker:authn-arch-api-oidc

:ship: See on GHCR. Use by referencing with full name as printed above, mind the registry name.

Oct 03 '24 20:10 github-actions[bot]

:package: Pushed preview images as

ghcr.io/gdcc/dataverse:authn-arch-api-oidc

ghcr.io/gdcc/configbaker:authn-arch-api-oidc

:ship: See on GHCR. Use by referencing with full name as printed above, mind the registry name.

Oct 03 '24 20:10 github-actions[bot]

:package: Pushed preview images as

ghcr.io/gdcc/dataverse:authn-arch-api-oidc

ghcr.io/gdcc/configbaker:authn-arch-api-oidc

:ship: See on GHCR. Use by referencing with full name as printed above, mind the registry name.

Oct 03 '24 21:10 github-actions[bot]

:package: Pushed preview images as

ghcr.io/gdcc/dataverse:authn-arch-api-oidc

ghcr.io/gdcc/configbaker:authn-arch-api-oidc

:ship: See on GHCR. Use by referencing with full name as printed above, mind the registry name.

Oct 04 '24 11:10 github-actions[bot]

:package: Pushed preview images as

ghcr.io/gdcc/dataverse:authn-arch-api-oidc

ghcr.io/gdcc/configbaker:authn-arch-api-oidc

:ship: See on GHCR. Use by referencing with full name as printed above, mind the registry name.

Oct 04 '24 13:10 github-actions[bot]

:package: Pushed preview images as

ghcr.io/gdcc/dataverse:authn-arch-api-oidc

ghcr.io/gdcc/configbaker:authn-arch-api-oidc

:ship: See on GHCR. Use by referencing with full name as printed above, mind the registry name.

Oct 04 '24 16:10 github-actions[bot]

:package: Pushed preview images as

ghcr.io/gdcc/dataverse:authn-arch-api-oidc

ghcr.io/gdcc/configbaker:authn-arch-api-oidc

:ship: See on GHCR. Use by referencing with full name as printed above, mind the registry name.

Oct 04 '24 20:10 github-actions[bot]

I had now reimplemented the entire OIDC code with the new mechanism. In the process I had to drop the PKCE support, which makes sense i.m.o. since the entire OIDC is now handled by the Payara built-in feature and the frontends do not need to implement it anymore. Another unexpected outcome is that I had to reimplement the bearer token support. I feel like dropping it would make this PR even harder to accept. The final implementation is not extremely bad, it reuses the cache that was first meant for the PKCE support. However, it makes the bearer tokens being stored twice in the system: once by the sessions, and once by the cache for the bearer support. I still believe that not storing the cookies, but generating one with each request is not much different from using the bearer tokens (I would leave the bearer token support feature off). For example, when using curl and not storing the cookie, the requests look like this:

curl -v --cookie "JSESSIONID=short-session-id" http://localhost:8080/api/v1/users/:me | jq .

vs

curl -H "Authorization: Bearer very-long-bearer-token-encoded-in-base64" http://localhost:8080/api/v1/users/:me | jq .

I still need to reimplement the "first login" feature for the unknown users (when email address is not found in the DB) and debug the code. Nevertheless, the main ideas are already in the code.

Oct 04 '24 20:10 ErykKul

:package: Pushed preview images as

ghcr.io/gdcc/dataverse:authn-arch-api-oidc

ghcr.io/gdcc/configbaker:authn-arch-api-oidc

:ship: See on GHCR. Use by referencing with full name as printed above, mind the registry name.

Oct 04 '24 22:10 github-actions[bot]

:package: Pushed preview images as

ghcr.io/gdcc/dataverse:authn-arch-api-oidc

ghcr.io/gdcc/configbaker:authn-arch-api-oidc

:ship: See on GHCR. Use by referencing with full name as printed above, mind the registry name.

Oct 05 '24 00:10 github-actions[bot]

:package: Pushed preview images as

ghcr.io/gdcc/dataverse:authn-arch-api-oidc

ghcr.io/gdcc/configbaker:authn-arch-api-oidc

:ship: See on GHCR. Use by referencing with full name as printed above, mind the registry name.

Oct 05 '24 16:10 github-actions[bot]

:package: Pushed preview images as

ghcr.io/gdcc/dataverse:authn-arch-api-oidc

ghcr.io/gdcc/configbaker:authn-arch-api-oidc

:ship: See on GHCR. Use by referencing with full name as printed above, mind the registry name.

Oct 05 '24 19:10 github-actions[bot]

:package: Pushed preview images as

ghcr.io/gdcc/dataverse:authn-arch-api-oidc

ghcr.io/gdcc/configbaker:authn-arch-api-oidc

:ship: See on GHCR. Use by referencing with full name as printed above, mind the registry name.

Oct 05 '24 21:10 github-actions[bot]

@pdurbin I have made a lot of progress on this. It is very clean now, looks like it works very well. I reimplemented the bearer token again, I got this right now, I believe (no more storing of the tokens in the DB or caffeine cache and the code is unbelievably short). The coolest part is the API bearer token test from python (see also the updated test procedure):

cd bearer-token-example
./run.sh

At this point I do not believe we can do much better than this. All of the heavy-lifting is outsourced to the payara. The only feature that is gone is the PKCE support, but I believe that it is a good thing. The implementation is also multi-tenant and you can use multiple OIDC configurations in one dataverse instance (see also the payara doc https://docs.payara.fish/enterprise/docs/Technical%20Documentation/Public%20API/OpenID%20Connect%20Support.html). It has now more features than we could hope implementing :-D It even has auto-refreshing feature for the tokens.

@poikilotherm I have noticed this feature in OpenIdAuthenticationDefinition that can be used (not sure if it is helpful or if it will work on your systems, but it might be worth looking at it):

    /**
     * Optional. The prompt value specifies whether the authorization server
     * prompts the user for reauthentication and consent. If no value is
     * specified and the user has not previously authorized access, then the
     * user is shown a consent screen.
     *
     * @return
     */
    PromptType[] prompt() default {};

Oct 05 '24 22:10 ErykKul

:package: Pushed preview images as

ghcr.io/gdcc/dataverse:authn-arch-api-oidc

ghcr.io/gdcc/configbaker:authn-arch-api-oidc

:ship: See on GHCR. Use by referencing with full name as printed above, mind the registry name.

Oct 05 '24 22:10 github-actions[bot]

@ErykKul Thanks, this looks good. I'm back in the office on Thursday and will test this implementation.

Oct 07 '24 07:10 johannes-darms

:package: Pushed preview images as

ghcr.io/gdcc/dataverse:authn-arch-api-oidc

ghcr.io/gdcc/configbaker:authn-arch-api-oidc

:ship: See on GHCR. Use by referencing with full name as printed above, mind the registry name.

Oct 07 '24 10:10 github-actions[bot]

:package: Pushed preview images as

ghcr.io/gdcc/dataverse:authn-arch-api-oidc

ghcr.io/gdcc/configbaker:authn-arch-api-oidc

:ship: See on GHCR. Use by referencing with full name as printed above, mind the registry name.

Oct 07 '24 11:10 github-actions[bot]

:package: Pushed preview images as

ghcr.io/gdcc/dataverse:authn-arch-api-oidc

ghcr.io/gdcc/configbaker:authn-arch-api-oidc

:ship: See on GHCR. Use by referencing with full name as printed above, mind the registry name.

Oct 07 '24 11:10 github-actions[bot]

:package: Pushed preview images as

ghcr.io/gdcc/dataverse:authn-arch-api-oidc

ghcr.io/gdcc/configbaker:authn-arch-api-oidc

:ship: See on GHCR. Use by referencing with full name as printed above, mind the registry name.

Oct 07 '24 11:10 github-actions[bot]

@ErykKul is this ready to review? I'm halfway through a review but on a moving target... :wink:

Oct 07 '24 11:10 poikilotherm