Bug Report: Deployment to container fails with java.lang.NoClassDefFoundError: fish/payara/logging/PayaraLogManager

[zkr-bp]

Description

---

Hi,

we've recently upgraded the payara version of the base image we use to pack our .war into when running integration tests with testcontainers and we are getting a NoClassDefFoundError.

We've previously used: payara/server-full:5.2020.6-jdk11

Now we've upgraded to: payara/server-full:5.2021.4-jdk11

Expected Outcome

War should be deployed.

Current Outcome

We get this exception: [#|2022-06-07T08:13:37.181+0000|SEVERE|Payara 5.2021.4|javax.enterprise.web.core|_ThreadID=1;_ThreadName=main;_TimeMillis=1654589617181;_LevelValue=1000;| Error invoking ServletContainerInitializer org.springframework.web.SpringServletContainerInitializer java.lang.NoClassDefFoundError: fish/payara/logging/PayaraLogManager at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.<clinit>(GlassFishMain.java:79) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:315) at org.springframework.boot.SpringApplication.deduceMainApplicationClass(SpringApplication.java:272) at org.springframework.boot.SpringApplication.<init>(SpringApplication.java:264) at org.springframework.boot.builder.SpringApplicationBuilder.createSpringApplication(SpringApplicationBuilder.java:129) at org.springframework.boot.builder.SpringApplicationBuilder.<init>(SpringApplicationBuilder.java:102) at org.springframework.boot.builder.SpringApplicationBuilder.<init>(SpringApplicationBuilder.java:98) at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.createSpringApplicationBuilder(SpringBootServletInitializer.java:166) at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.createRootApplicationContext(SpringBootServletInitializer.java:130) at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.onStartup(SpringBootServletInitializer.java:97) at org.springframework.web.SpringServletContainerInitializer.onStartup(SpringServletContainerInitializer.java:174) at org.apache.catalina.core.StandardContext.callServletContainerInitializers(StandardContext.java:5855) at com.sun.enterprise.web.WebModule.callServletContainerInitializers(WebModule.java:736) at org.apache.catalina.core.StandardContext.start(StandardContext.java:5753) at com.sun.enterprise.web.WebModule.start(WebModule.java:619) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:958) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:941) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:694) at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:1851) at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:1603) at com.sun.enterprise.web.WebApplication.start(WebApplication.java:108) at org.glassfish.internal.data.EngineRef.start(EngineRef.java:123) at org.glassfish.internal.data.ModuleInfo.start(ModuleInfo.java:293) at org.glassfish.internal.data.ApplicationInfo.start(ApplicationInfo.java:364) at com.sun.enterprise.v3.server.ApplicationLifecycle.initialize(ApplicationLifecycle.java:623) at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:580) at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:556) at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:552) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/javax.security.auth.Subject.doAs(Subject.java:361) at com.sun.enterprise.v3.admin.CommandRunnerImpl$2.execute(CommandRunnerImpl.java:551) at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:582) at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:574) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/javax.security.auth.Subject.doAs(Subject.java:361) at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:573) at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1497) at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1300(CommandRunnerImpl.java:120) at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1879) at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1755) at com.sun.enterprise.admin.cli.embeddable.CommandExecutorImpl.executeCommand(CommandExecutorImpl.java:183) at com.sun.enterprise.admin.cli.embeddable.CommandExecutorImpl.run(CommandExecutorImpl.java:96) at fish.payara.boot.runtime.BootCommand.execute(BootCommand.java:69) at fish.payara.boot.runtime.BootCommands.executeCommands(BootCommands.java:136) at fish.payara.boot.runtime.BootCommands.executeCommands(BootCommands.java:130) at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.doBootCommands(GlassFishMain.java:304) at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMain.java:143) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:119) at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:54) Caused by: java.lang.ClassNotFoundException: fish.payara.logging.PayaraLogManager not found by fish.payara.server.internal.core.glassfish [50] at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1597) at org.apache.felix.framework.BundleWiringImpl.access$300(BundleWiringImpl.java:79) at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1982) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ... 54 more |#]

Steps to reproduce

Create docker image with payara/server-full:5.2021.4-jdk11 as base image, deploy a war (a simple spring boot hello world application would do).

Environment

• Distribution: payara/server-full:5.2021.4-jdk11
• JDK Version: zulu11.45.27-ca-jdk11.0.10-win_x64
• Operating System: Windows

[helmutsiegel]

I have the same problem with the same version of payara.

[JamesHillyard]

Hi @zkr-bp, @helmutsiegel

Could you please verify your issue on the latest version of Payara Community edition and confirm if this error still occurs, currently that is 5.2022.2. The JDK-11 docker image, payara/server-full:5.2022.2-jdk11 exists for the latest version of Payara Community.

Thanks, James

[zkr-bp]

Hi, @JamesHillyard,

yes, the issue persists with payara/server-full:5.2022.2-jdk11.

[JamesHillyard]

Hi @zkr-bp,

Thank you for confirming the issue on the latest version of Payara Community Edition. Using my own application on the payara/server-full:5.2022.2-jdk11 docker image didn't result in the same error you are receiving.

Please, can you provide the code you are using or a simple to follow scenario including sample code to reproduce this on the latest release of Payara Community Edition? A reproducer should ideally follow the SSCCE rules: http://www.sscce.org/. It will greatly help us to find the cause and fix it.

Thanks, James

[zkr-bp]

Using my own application on the payara/server-full:5.2022.2-jdk11 docker image didn't result in the same error you are receiving.

I have a hunch it has something to do with Spring Boot. Was yours a Spring Boot application?

[JamesHillyard]

Hi @zkr-bp,

I have tried with both a simple hello world spring application and a standard Jakarta EE application, neither produced the error you are getting. Could you please provide your simple reproducer application so I can compare it to mine and try reproduce the error?

Thanks, James

[zkr-bp]

Hi @JamesHillyard, created a repo for you: https://github.com/zkr-bp/demo

[JamesHillyard]

Hi @zkr-bp,

When going to deploy your reproducer I was faced with the following error: java.lang.NoClassDefFoundError: org/springframework/batch/core/configuration/annotation/BatchConfigurer which is not the same as the error you reported.

The fix to the NoClassDefFoundError I enountered, I removed the unused dependency:

<dependency>
	<groupId>org.springframework</groupId>
	<artifactId>spring-web</artifactId>
	<version>5.3.1</version>
</dependency>

After that I was able to deploy the application successfully. I'll list the steps I took to reproduce the issue, if this differs to your reproducing method, please do let me know:

1. Clone the reproducer at https://github.com/zkr-bp/demo
2. Run mvn clean install on the reproducer
3. Execute docker run -p 8080:8080 -p 4848:4848 payara/server-full:5.2022.2-jdk11
4. Open the admin console and deploy demo-0.0.1-SNAPSHOT.war
5. Deployment successful

Environment: Windows 10 Maven 3.6.3 Azul Zulu JDK 11.0.11+9 Payara server-full:5.2022.2-jdk11

Thanks, James

[JamesHillyard]

Hi @zkr-bp,

Were you able to fix this issue? If not could you please provide an update to the reproducer which will allow me to reproduce the error you are encountering locally.

Thanks, James

[JamesHillyard]

Hi @zkr-bp,

Since we haven’t received any updates from you, we’ll proceed to close this issue. Feel free to re-open it in the future if you are able to reproduce the problem successfully using the current release of Payara Community Edition at that point in time.

Thanks, James