dataverse Spike: Upgrade to Payara 6 and from EE 8 to EE 10 (end of Payara 5 Community Edition security patches Q2/2022)

Hat tip to @poikilotherm who posted in chat a link to a video in which Steve Millidge, founder of Payara, answered a question about how long Payara 5 Community Edition (which we recommend in the guides) will receive security patches: https://youtu.be/juD0XNNq344?t=1972

I've transcribed the question and answer:

Q: "For the Community Edition, when will v5 stop getting security patches?"

A: "Payara 5 in Community was basically... when we switch the main branch over to 6, which will be on the point of 6 release, then Community will march forward on Payara 6. So Payara 5 patches, if you like, will stop at that point through Community. So then it will be Enterprise where you'll get patches and security fixes for Payara 5. Payara Community essentially is a marching version so whenever the latest release is the latest release."

I didn't watch the whole video but @poikilotherm says Payara plans to switch the main branch to 6 in Q2/2022.

(By the way, #8064 is the issue where we're tracking the next 5.x upgrade from Payara. As of this writing we are advising installations to use Payara 5.2021.5: https://guides.dataverse.org/en/5.9/installation/prerequisites.html#payara )

Dec 10 '21 15:12 pdurbin

At https://youtu.be/Bm8aNy2bcJ4?t=1121 some dates were shown for Payara 6. Here' are some screenshots:

Feb 01 '22 20:02 pdurbin

Payara Server Community 6.2022.1 Alpha 2 is available for download from https://www.payara.fish/downloads/payara-platform-community-edition/

Based on https://github.com/payara/Payara/releases/tag/payara-server-6.2022.1.Alpha2 it looks like it came out on Feb 17.

Mar 08 '22 17:03 pdurbin

Current state of develop on payara-6.2022.1.Alpha2 (openjdk 11.0.14):

[INFO] Building war: /tmp/dataverse/target/dataverse-5.9.war
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

Deploying the application (dataverse.war)
Failed to deploy the application!

5,341 "Invalid signature file digest" SEVEREs, then:

org.xml.sax.SAXParseExceptionpublicId: file:/usr/local/payara6/glassfish/lib/schemas/web-fragment_3_1.xsd; lineNumber: 8; columnNumber: 27; Deployment descriptor file META-INF/web-fragment.xml in archive [omnifaces-3.8.jar].  TargetNamespace.1: Expecting namespace 'https://jakarta.ee/xml/ns/jakartaee', but the target namespace of the schema document is 'http://xmlns.jcp.org/xml/ns/javaee'
org.xml.sax.SAXParseException; systemId: file:/usr/local/payara6/glassfish/lib/schemas/web-fragment_3_1.xsd; lineNumber: 8; columnNumber: 27; TargetNamespace.1: Expecting namespace 'https://jakarta.ee/xml/ns/jakartaee', but the target namespace of the schema document is 'http://xmlns.jcp.org/xml/ns/javaee'

Mar 09 '22 13:03 donsizemore

@pdurbin asked for the commit (1487650d44a308718d15b2e8c67fdec5422f0971) and full server.log (attached as payara6_server_log.zip )

Mar 09 '22 17:03 donsizemore

I changed all instances of http://xmlns.jcp.org/xml/ns/javaee to https://jakarta.ee/xml/ns/jakartaee in https://github.com/OdumInstitute/dataverse/commit/91ef3264918cbf6312b7b8965405e9bf363b3421 but still get the namespace error on deployment.

@pdurbin notes the omnifaces error, and suggests an upgrade. OmniFaces 3.13 produces the same namespace error on deployment; omnifaces.org tells me "use 4.0-M13 instead. It’s the Jakartified version of 3.13."

A mvn package using OmniFaces 4.0.M-13 yields

[ERROR] /tmp/dataverse/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/OAuth2LoginBackingBean.java:[89,50] cannot access jakarta.servlet.http.HttpServletRequest
  class file for jakarta.servlet.http.HttpServletRequest not found

I suppose I could drop back to OmniFaces 4.0.M-3 "(25 Oct 2020, Jakartified 3.8.1, still using Java 1.8)" but this seems wrong to me. Suggestions from smarter minds most welcomed.

Mar 14 '22 19:03 donsizemore

Current state: Jim has an EE9 branch and Oliver has a Jakarta branch

Oliver graciously used his commercial IntelliJ "migrate javax to jakarta" function, as a global search and replace isn't advised since not all javax packages will be migrated.
Jakarta namespace change will require an OmniFaces upgrade and possibly a commons-fileupload upgrade/replacement
Currently blocked by SWORD2 work, as the current SWORD implementation expects javax.*

One thing to note or to ignore: as of Nov 2021 Payara runs on the 17 LTS JDK.

Mar 23 '22 18:03 donsizemore

In this upgrade, we need to see if https://github.com/IQSS/dataverse/issues/8064#issuecomment-965627984 is still relevant as an upgrade issue.

Mar 29 '22 10:03 qqmyers

Payara just announced 5.2022.2 with the following remark:

"Please note: This is the penultimate Payara 5 Community release. Payara 6 Community will soon take its place, to be used with Jakarta EE 10. If you want to keep using earlier Java EE/Jakarta EE versions – we encourage you to move to Payara 5 Enterprise."

AFAIK, JDK 17 is not a must for Jakarta EE 10, but would be recommended.

Apr 21 '22 15:04 Kris-LIBIS

Sprint

The free version we are on is approaching EOL.
Don tried an upgrade and it did not work initially.
Jim suggests starting with oliver to see where he's gotten to. In Jakarta there were path changes.
make this a Spike. This is not alot of unknown, but will be a lot of looking at paths.

Large. Because no one has any idea of size, but we need to get through it.

Apr 27 '22 19:04 mreekie

I just updated my Jakarta EE 9.1 branch to latest develop.

Things to do and done already:

[X] Migrate to Jakarta Namespace
[X] Migrate to Primefaces Jakarta EE 9+ variant
[X] Update JSONLD library to a Jakarta EE 9+ based version
[X] Migrate web.xml, faces config and others to Jakarta Faces 3.0 (will need more changes for 4.0 in EE 10!)
[X] Migrate JMS class names string references to Jakarta EE 9+ ones
[X] Migrate persistance.xml to be compliant with Jakarta EE 9
[X] Migrate beans.xml to Jakarta EE 9 version
[X] Make use of a Jakartified SWORD2 library snapshot release
[ ] Migrate to Jakartified SWORD2 library release (no snapshots in develop or master!)
[ ] Migrate to a Jakartified XOAI lib (which needs and receives refactoring anyway, see #8372)

As there is a magic rewriter for the namespace in Payara, you can still try this with the old XOAI lib and potentially others not using Jakarta already.

For a more details list of changes, take a look at compare. I invested time into meaningful commit messages.

Apr 28 '22 16:04 poikilotherm

“Payara Platform Community will move to Payara 6 and Jakarta EE 10 at the end of May 2022.” https://www.payara.fish/what-payara-6-means-for-you/

May 11 '22 20:05 mreekie

Sprint -

This has to be done by the end of may.
It really can only be done by 1 person. (can't be split at least initially)

May 11 '22 20:05 mreekie

Just a heads up: I pushed the most recent develop with Jakarta conversion to my branch https://github.com/poikilotherm/dataverse/tree/jakarta

To update, you should use git pull --rebase instead of the usual merge.

It now contains the Primefaces 11 change, too. It does, however, not use EE 10 - it's still on EE 9.1!

May 12 '22 12:05 poikilotherm

in WIP at end of pm.sprint.2022_05_11

May 25 '22 17:05 mreekie

@mreekie mentioned that we aren't always good about commenting with updates. It's true! The info tends to go into Slack and meetings (standup and tech hours).

The main thing I wanted to mention is that I'm closing the following issue:

#8714

We have a workaround in place as of d37af1d. In short @rtreacy hacked on the Rewrite/PrettyFaces library until it could deploy to Payara 6. I tweaked it a bit and pushed the code to https://github.com/pdurbin/rewrite/commit/d494160 , built the jars we need, and added them to the code base (again in d37af1d).

I checked with @poikilotherm today and he's on board with forking Rewrite to the @gdcc GitHub org instead and releasing to gdcc's Maven Central namespace. ("Sure. That's an easy thing to do. We only need to rename the groupId and should be good to go.") This will allow us to remove the jars from local_lib (again, added in d37af1d).

A stretch goal is to get these changes into upstream Rewrite and that conversation is happening here: https://github.com/ocpsoft/rewrite/issues/304

As to the rest of this effort, here's the good news:

We can now deploy to Payara 6 (Python installer updated, docker-aio, Vagrant, ec2/Ansible).
PrettyFaces/Rewrite "pretty" URLs seem to be working fine.

Here's the bad news:

The SWORD API isn't working.
Most API tests are failing (some due to SWORD failures)
We've only been testing namespace changes (EE 9.1) but EE 10 changes are coming. We can't test this until Payara 6 Alpha 3 is out.

Here's some stuff we've talked about doing:

Fix SWORD
Get the API tests passing
Test EE 10 when Alpha 3 is out
If possible, make smaller pull requests against develop
Update the guides (I took a pass at the dev guide already)
Set up a Payara 6 test server for the community to bang on, like we did for Payara 5: https://groups.google.com/g/dataverse-dev/c/5hfiXl4Hpd8/m/awzoSpupAQAJ
Release rewrite under GDCC Maven and remove the versions from local_lib
Update Perl installer (Python installer should be all set)
Once Payara 6 final is out, re-test, update URLs in code and docs.

May 25 '22 20:05 pdurbin

In 6f54d1ffb5 I fixed the SWORD APIs by reverting f7234c41fc in the Payara 6 branch that added bean-discovery-mode="annotated". I changed it to all, at least for now. My understanding is that when we get Payara 6 Alpha 3 and EE 10 instead of EE 9.1, we will have to switch it back to annotated and figure out what annotations to add in various places (hopefully only SWORD code but who knows).

Then I noticed that uploading files to SWORD was failing like this:

curl -u 46abc733-e454-4f7f-b498-aeec5815de4f: --data-binary @path/to/example.zip -H "Content-Disposition: filename=example.zip" -H "Content-Type: application/zip" -H "Packaging: http://purl.org/net/sword/package/SimpleZip" http://localhost:8080/dvn/api/data-deposit/v1.1/swordv2/edit-media/study/doi:10.5072/FK2/MJQYQ0

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<sword:error xmlns="http://www.w3.org/2005/Atom" href="http://purl.org/net/sword/error/ErrorBadRequest" xmlns:sword="http://purl.org/net/sword/terms/">
  <title>ERROR</title>
  <updated>2022-05-27T19:55:29Z</updated>
  <generator uri="http://www.swordapp.org/" version="2.0"/>
  <summary>Filename could not be extracted from Content-Disposition: Expected separator ';' instead of '='</summary>
  <sword:treatment>Processing failed</sword:treatment>
</sword:error>

The fix, apparently, is to change...

-H "Content-Disposition: filename=example.zip"

... to...

-H "Content-Disposition: attachment; filename=example.zip"

Unfortunately, this is a backward-incompatible change. It would nice to figure out what's going on and if we can preserve the old behavior for backward compatibility. For now I updated the API tests, the SWORD section of the API Guide, and the release note in d86e2204c9

May 27 '22 20:05 pdurbin

As of d86e2204c9 (but not because of that commit), I'm getting the following stacetrace from EditDDIIT:

[2022-06-03T12:25:43.649-0400] [Payara 6.2022.1.Alpha2] [WARNING] [AS-EJB-00056] [jakarta.enterprise.ejb.container] [tid: _ThreadID=113 _ThreadName=http-thread-pool::http-listener-1(2)] [timeMillis: 1654273543649] [levelValue: 900] [[ A system exception occurred during an invocation on EJB EditDDI, method: public jakarta.ws.rs.core.Response edu.harvard.iq.dataverse.api.EditDDI.edit(java.io.InputStream,java.lang.String)]]

[2022-06-03T12:25:43.649-0400] [Payara 6.2022.1.Alpha2] [WARNING] [] [jakarta.enterprise.ejb.container] [tid: _ThreadID=113 _ThreadName=http-thread-pool::http-listener-1(2)] [timeMillis: 1654273543649] [levelValue: 900] [[

jakarta.ejb.EJBException ... Caused by: java.lang.NullPointerException at org.glassfish.grizzly.http.server.Request.getRemoteAddr(Request.java:1132) at org.apache.catalina.connector.Request.getRemoteAddr(Request.java:1453) at org.apache.catalina.connector.RequestFacade.getRemoteAddr(RequestFacade.java:531) at jakarta.servlet.ServletRequestWrapper.getRemoteAddr(ServletRequestWrapper.java:222) at jakarta.servlet.ServletRequestWrapper.getRemoteAddr(ServletRequestWrapper.java:222) at edu.harvard.iq.dataverse.engine.command.DataverseRequest.(DataverseRequest.java:126) at edu.harvard.iq.dataverse.api.EditDDI.createNewDraftVersion(EditDDI.java:192) at edu.harvard.iq.dataverse.api.EditDDI.edit(EditDDI.java:136)

Furthermore, I was seeing the following stacktrace on the testImportDDI method of DataversesIT:

[2022-06-03T12:33:04.254-0400] [Payara 6.2022.1.Alpha2] [WARNING] [] [org.eclipse.persistence.session./file:/usr/local/payara6/glassfish/domains/domain1/applications/dataverse-5.10.1/WEB-INF/classes/_VDCNet-ejbPU] [tid: _ThreadID=114 _ThreadName=http-thread-pool::http-listener-1(3)] [timeMillis: 1654273984254] [levelValue: 900] [[

Local Exception Stack: Exception [EclipseLink-5003] (Eclipse Persistence Services - 3.0.2.v202107160933): org.eclipse.persistence.exceptions.OptimisticLockException Exception Description: The object [[DatasetVersion id:212]] cannot be deleted because it has changed or been deleted since it was last read. Class> edu.harvard.iq.dataverse.DatasetVersion Primary Key> 212 at org.eclipse.persistence.exceptions.OptimisticLockException.objectChangedSinceLastReadWhenDeleting(OptimisticLockException.java:134) ... at edu.harvard.iq.dataverse.EJB31_Generated__EjbDataverseEngine__Intf____Bean.submit(Unknown Source) at edu.harvard.iq.dataverse.api.AbstractApiBean.execCommand(AbstractApiBean.java:629) at edu.harvard.iq.dataverse.api.Datasets.lambda$destroyDataset$2(Datasets.java:347) at edu.harvard.iq.dataverse.api.AbstractApiBean.response(AbstractApiBean.java:700) at edu.harvard.iq.dataverse.api.Datasets.destroyDataset(Datasets.java:334)

I fixed these errors in 8be36a354e by removing @Context protected HttpServletRequest httpRequest (switching to our standard createDataverseRequest instead) and adding a sleep.

Jun 03 '22 17:06 pdurbin

Here's an update on the items above:

~~Fix SWORD~~ - DONE but added a note in the PR to think more about preserving backward compatibility.
Get the API tests passing - Made pull request #8774 to find out (Update: whoops, this won't work yet but we have a dedicated job to look at here: https://jenkins.dataverse.org/job/IQSS-Dataverse-Payara6/ )
Test EE 10 when Alpha 3 is out - BLOCKED, Alpha 3 is not out yet: https://github.com/payara/Payara/releases
~~If possible, make smaller pull requests against develop~~ - nothing obvious
~~Update the guides (I took a pass at the dev guide already)~~ - DONE
Set up a Payara 6 test server for the community to bang on, like we did for Payara 5: https://groups.google.com/g/dataverse-dev/c/5hfiXl4Hpd8/m/awzoSpupAQAJ - Don is working on this
Release rewrite under GDCC Maven and remove the versions from local_lib - need to talk to Oliver about this. I did fork the project to https://github.com/gdcc/rewrite and PR https://github.com/ocpsoft/rewrite/pull/322 is from that fork. I've been asked to make some changes to that PR.
Update Perl installer (Python installer should be all set) - need to touch base with Leonid on this
Once Payara 6 final is out, re-test, update URLs in code and docs - BLOCKED, again Alpha 2 is not out yet

As is our custom, I'll remove this issue from the board and use the pull request for updates:

#8774

Jun 03 '22 19:06 pdurbin

@poikilotherm 's most recent commits package and test cleanly on payara-6.2022.1.Alpha3, but the warfile doesn't deploy. I have the full server.log if you like, but in a nutshell:

$ grep NoClass /usr/local/payara6/glassfish/domains/domain1/logs/server.log
java.lang.NoClassDefFoundError: jakarta/faces/webapp/UIComponentELTag
org.apache.catalina.LifecycleException: org.apache.catalina.LifecycleException: java.lang.NoClassDefFoundError: jakarta/faces/webapp/UIComponentELTag
Caused by: org.apache.catalina.LifecycleException: java.lang.NoClassDefFoundError: jakarta/faces/webapp/UIComponentELTag
Caused by: java.lang.NoClassDefFoundError: jakarta/faces/webapp/UIComponentELTag
  java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: org.apache.catalina.LifecycleException: java.lang.NoClassDefFoundError: jakarta/faces/webapp/UIComponentELTag
java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: org.apache.catalina.LifecycleException: java.lang.NoClassDefFoundError: jakarta/faces/webapp/UIComponentELTag
java.lang.Exception: java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: org.apache.catalina.LifecycleException: java.lang.NoClassDefFoundError: jakarta/faces/webapp/UIComponentELTag
java.lang.Exception: java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: org.apache.catalina.LifecycleException: java.lang.NoClassDefFoundError: jakarta/faces/webapp/UIComponentELTag
  Exception while loading the app : java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: org.apache.catalina.LifecycleException: java.lang.NoClassDefFoundError: jakarta/faces/webapp/UIComponentELTag]]

Jul 20 '22 18:07 donsizemore

I dug into this a bit. That class has been removed from Jakarta Faces 4 because of dropping support for JSP in Faces.

See jakartaee/faces#1546, eclipse-ee4j/mojarra#4775, eclipse-ee4j/mojarra#4777

I know that the Payara Admin GUI is broken (as is in GF7, discussed in eclipse-ee4j/glassfish#23782) due to these changes. Now I'm not sure where things fall apart in our app.

@donsizemore could you add the full server log to pastebin or upload here as file? Would be good to look at more details of where this fails. Thanks!

Jul 21 '22 06:07 poikilotherm

@poikilotherm good catch. I do see UIComponentELTag being removed in this PR:

https://github.com/eclipse-ee4j/mojarra/pull/4777

I'm surprised we're using JSP in Dataverse. I thought we were all JSF.

Jul 21 '22 11:07 pdurbin

@poikilotherm payara6_server.log

Jul 21 '22 14:07 donsizemore

Payara Community 6.2022.1 Alpha 4 is out: https://docs.payara.fish/community/docs/6.2022.1.alpha4/Release%20Notes/Release%20Notes%206.2022.1.Alpha4.html

Sep 20 '22 11:09 donsizemore

During tech hour yesterday the devs agreed we probably should freeze develop at some point in the future (maybe that point could be a 5.13 release) and do the breaking changes then. Every feature PR from then on needs to incorporate the necessary changes for the Jakarta EE 10 transition.

In case of an urgency, we can still create some point release based on the frozen status, as this would be unlikely to introduce lots of new classes or usage of Java EE instead of Jakarta EE.

We also agreed it makes a lot of sense to have a release of Payara 6 available before we ramp up team resources on testing and ironing out any bugs and issues on the way. When, how and with what human resources this transition will happen is to be discussed soon.

Nov 02 '22 08:11 poikilotherm

"Payara Platform Community 5.2022.5 is the final release of the Payara 5 Community stream... After this it will no longer be maintained; there will be no more bug fixes, component upgrades or improvements. In short, it will no longer be safe to use. -- https://blog.payara.fish/whats-new-in-the-december-2022-payara-platform-release

Dec 13 '22 16:12 pdurbin

The final release is out today: https://github.com/payara/Payara/releases/tag/payara-server-5.2022.5

Dec 13 '22 16:12 poikilotherm

Next steps:

#9283

Jan 17 '23 16:01 pdurbin

Today I tested the Payara 6 branch a bit (#9116) and left a comment here with some findings: https://github.com/IQSS/dataverse/issues/9283#issuecomment-1403927611

Like @poikilotherm said above, the final version of Payara 5 community edition came out in mid December 2022.

From a sprint kickoff meeting today it sounds like the plan is:

release Dataverse 5.13 soonish (runs on Payara 5)
release Dataverse 5.14 in April or so (runs on Payara 5), get some GDCC work in, especially. See the milestone: https://github.com/IQSS/dataverse/milestone/108
release a Dataverse version (6.0 perhaps) that runs on Payara 6 in early June or so, before the community meeting

Jan 26 '23 02:01 pdurbin

As this means plenty of time how about making it Payara 6 + Java 17?

Jan 26 '23 07:01 poikilotherm

@poikilotherm hmm, worth a try, I'd say. You've made a lot of progress your PR:

#9291

I see the Payara 6 upgrade as a security issue so I wouldn't want to hold it up for Java 17, if Java 17 gives us trouble.

While I'm leaving a comment anyway, we talked about this yesterday but we should make sure API tests are passing. I just created an issue for this:

#9329

Jan 26 '23 12:01 pdurbin