camunda-bpm-platform icon indicating copy to clipboard operation
camunda-bpm-platform copied to clipboard

Published 20 hours ago • verified publisher icon camunda

Support Tomcat 10

Open ThorbenLindhauer opened this issue 3 years ago • 18 comments

This issue was imported from JIRA:

Field Value
JIRA Link CAM-13275
Reporter @ThorbenLindhauer
Has restricted visibility comments false

User Story (Required on creation):

Tomcat 10 was released in February 2021: https://tomcat.apache.org/download-10.cgi. Tomcat is a popular environment among our users.

Functional Requirements (Required before implementation):

  • Tomcat 10 is a supported environment for shared engine and webapps
  • We need to backport this for 7.21; since we need to provide Tomcat 9 and 10 support for 7.21, this is a good reason to keep it as well for 7.22
    • There are still many users relying on Tomcat 9 and Java EE. Java EE is not available on Tomcat 10.

Technical Requirements (Required before implementation):

  • There is sufficient CI for Tomcat 10

Limitations of Scope (Optional):

Hints (Optional):

### Tasks
- [ ] https://github.com/camunda/camunda-bpm-platform/issues/4386
- [ ] https://github.com/camunda/camunda-bpm-platform/issues/4391
- [ ] https://github.com/camunda/camunda-bpm-platform/issues/4440
- [ ] https://github.com/camunda/camunda-bpm-platform/issues/4434
- [ ] https://github.com/camunda/camunda-bpm-platform/issues/4435

Links:

  • is related to https://jira.camunda.com/browse/SUPPORT-13780
  • is related to https://jira.camunda.com/browse/SUPPORT-19605

Dev2QA handover

In order to test the Support of Tomcat 10, the QA can download the camunda-bpm-tomcat-7.22.0.zip distro and use the invoice example (or any other of your choice) to test all flows such as cockpit, tasklist, deployment, start process instances etc, with a db of their choice.

ThorbenLindhauer avatar Mar 16 '21 13:03 ThorbenLindhauer

This comment was imported from JIRA and written by user vrJNmVd

What is this name? This pseudonym name was generated based on the user name in JIRA to protect the personal data of our JIRA users. You can use this identifier to search for issues by the same reporter.

Are there any plans to support Tomcat 10 / Servlet 5.0 in the near future?

ThorbenLindhauer avatar May 31 '22 14:05 ThorbenLindhauer

With Camunda 7.20.0 support for Spring Boot 2.x (https://docs.camunda.org/enterprise/announcement/#camunda-platform-7-20) ends. Spring Boot 3.x uses the jakarte namespace (Java EE 10) which does not run on Tomcat 9.x. But there is still no support for Tomcat 10 (https://docs.camunda.org/manual/7.20/introduction/supported-environments/#container-managed-process-engine-and-camunda-cockpit-tasklist-admin 1).

How can I run Camunda 7.20.0 with Spring Boot 3.x if I am forced to use Tomcat? Is there a solution for this setup?

This comment was copied from the forum (https://forum.camunda.io/t/tomcat-10-x/33031/3?u=hizr)

hizr avatar Oct 26 '23 04:10 hizr

Hi @hizr,

Our Spring Boot 3 starters come with embedded Tomcat and those use Tomcat 10. Note that this is also the only supported deployment scenario for our Spring Boot starters. We are not targeting to support Spring Boot 3 deployments on standalone Tomcats anytime in the future. The docs that you reference are for the shared engine installations where we currently only support Tomcat 9 (and may in the future move to Tomcat 10). What you can do in any case is of course to just see if your setup works with Tomcat 10 without being officially supported, it might well be. I understand that this is also what you have been doing with Tomcat 9/Spring Boot 2 up to this point.

ThorbenLindhauer avatar Oct 26 '23 07:10 ThorbenLindhauer

Kickoff

Context

Tomcat 10 is a supported environment.

Business Value

The supported environments of Camunda 7 is going to continue its evolution along with the newer updates of Tomcat container.

Tomcat is a popular environment for our users, its ongoing support is expected from our community. Besides the value of catching up with the ecosystem updates, our users can benefit from the new Tomcat version:

  • Security fixes
  • Performance enhancements & Optimisations
  • Updated Dependencies (Logging frameworks & Library compatibility)
  • Bug fixes & stabilities
  • Enhanced HTTP/2 Support
  • Deprecation and Removal of Legacy APIs
  • Package Renaming & support for the new jakarta namespace
  • Tomcat 10 officially supports all java versions compatible with the Jakarta EE 9 platform (Java 8+)
  • Servlet and JSP Specifications (Servlet 5.0, JSP 3.0)

Customer Requirements

As a user, i want to be able to download the distribution of Camunda 7 which is based on Tomcat 10 while i'm still able to use Tomcat 9.

Backwards compatibility

  • Q: Will users still be able to use Tomcat 9?

  • A: Yes, Tomcat 9 will continue to be supported so users that are currently using it will be able to continue to do so.

  • Q: Does this feature break backwards compatibility?

  • A: No, Tomcat 10 is a newly supported environment. However, if users want to migrate from an older Tomcat version to Tomcat 10 and have custom code that relies on the old namespace prior to the jakarta, there might be conflicts which require a migration. The migration is out of scope for the needs of this feature.

We can still prompt our users to the official source for migrating Tomcat 9 to Tomcat 10. Also, We need to stress in the Migration Guide 7.21 to 7.22 that a) The Docker Image and main Tomcat distribution of 7.22 will be Tomcat 10 b) Users need to migrate their Web applications and Java Plugins to the Jakarta Namespace.

  • Similar to how Wildfly 27 migration guide.
  • Here is the 7.18 to 7.19 Tomcat Update Page

Technical Solution Proposal

Currently, the Tomcat distribution has a dependency on the following components:

  • The camunda components:
    • camunda-engine-rest,
    • camunda-example-invoice,
    • camunda-webapp
  • Tomcat 9.0.85 runtime
  • The Resteasy 3.15.6.Final runtime - which is compatible with the javax namespace

Changes Required

The zero section is a prerequisite before moving into the manipulation of the tomcat distribution parts which bring the separate parts, assemble them and output the different distributions.

0. engine-rest needs to support a jakarta configuration for Tomcat
  • The assembly-jakarta module needs to use a similar web.xml config to initialise the Reasteasy servlets to the camunda-engine-rest module
    • Copy the web.xml from the assembly module
    • Use the Jakarta HttpServletDispatcher instead of HttpServlet30Dispatcher
    • Replace javax.ws.rs.Application servlet parameter with jakarta.ws.rs.Application; that will ensure the correct initialisation of the REST DefaultApplication such as REST handlers, config classes etc.
  • Add a new execution with id tomcat to the maven-assembly-plugin similar to the execution found in the assembly module.
  • Add a descriptor file called assembly-war-tomcat.xml to the assembly-jakarta module which is going to include the following:
    • engine-rest-core-jakarta and its transitive dependencies into the WEB-INF/lib folder
    • All the provided dependencies that that can be found in module engine-rest-core-jakarta. These include:
      • jakarta.servlet:jakarta.servlet-api
      • jakarta.activation:jakarta.activation-api
      • jakarta.ws.rs:jakarta.ws.rs-api:jar
      • jakarta.xml.bind:jakarta.xml.bind-api:jar
    • Don't forget to exclude test and source dependencies org.camunda.bpm:camunda-engine-rest-core-jakarta:jar:tests, org.camunda.bpm:camunda-engine-rest-core-jakarta:jar:sources
  • Configure the new tomcat execution to use the new descriptor

1. Tomcat module needs to assemble only Tomcat 10 distribution
1.1 Add a new assembly Module that assembles Tomcat 10
  • Use tomcat/assembly as a base to create a new module called assembly-jakarta
  • Its assembly.xml configuration needs to point to the latest Tomcat 10 property called version.tomcat
1.2 distro module needs use the assembly-jakarta module to output the Tomcat 10 distribution.
  • Include all the dependencies in the distro/tomcat/distro/pom.xml
    • Tomcat 10 dependencies: [camunda-engine-rest-jakarta, camunda-webapp-tomcat, camunda-example-invoice-jakarta]
1.3 distro module needs to use the dependencies and assemble the Tomcat 10 distro

Changes required in maven-assembly-plugin

  • Add the camunda-tomcat-assembly-jakarta dependency to the pom.xml to make it available to the assembly plugin

  • Create a new assembly-jakarta.xml which will use the jakarta artifacts; specifically:

    • camunda-tomcat-assembly-jakarta
    • camunda-engine-rest-jakarta
    • camunda-webapp-tomcat-jakarta <- The newly created module

  • Add a new execution configuration to the maven-assembly-plugin, named distro-tomcat10

    • It will use the newly created assembly-jakarta.xml
    • The output for tomcat 10 will be camunda-bpm-tomcat-${project.version}
1.4 Delete the camunda-tomcat-assembly Tomcat 9 module and make the new module the only ouput
  • Delete the camunda-tomcat-assembly
  • Rename the camunda-tomcat-assembly-jakarta module to camunda-tomcat-assembly
1.5 The tomcat/webapp module needs to assemble both webapp & webapp-jakarta configurations
  • Use tomcat/webapp as a base to create a new module called webapp-jakarta; name the module camunda-webapp-tomcat-jakarta
    • Replace the javax.ws.rs.Application parameter name with jakarta.ws.rs.Application; This will ensure that the servlet Applications for APIs (welcome, tasklist, engine, admin, cockpit) are initialised correctly.
  • Change the resteasy dependency from org.jboss.resteasy:resteasy-jaxrs to org.jboss.resteasy:resteasy-core:6.2.8.Final
    • This update is required in order for the Resteasy runtime to work with the new jakarta namespace

2. Tomcat Runtimes and QA Tests

After the migration of the property version.tomcat to point to the latest (tomcat 10 currently), the qa module will use the latest version for its tests when the tomcat profile is used which is desirable.

Both Tomcat 9 and Tomcat 10 versions will be tested against the CI to ensure coverage.

2.1 Make the existing Tomcat Runtime module of QA tests Tomcat 9 specific
  • 2.1.1 Rename the existing tomcat profile to tomcat9 under the qa module
  • 2.1.2 Rename the existing version.tomcat in tomcat.runtime.location to version.tomcat9
  • 2.1.3 Add a tomcat9 profile in the qa pom.xml similar to the existing tomcat profile
  • 2.1.4 Rename the existing tomcat profile to tomcat9 in the qa/integration-tests-engine module
  • 2.1.5 Make the module integration-tests-engine use the renamed dependency camunda-qa-tomcat9-runtime
2.2 Add a Tomcat 10 module & profile
  • 2.2.1 Add a tomcat-runtime module, similar to tomcat9-runtime
  • 2.2.2 Make it depend on property version.tomcat
  • 2.2.3 Adjust tomcat.runtime.location property to point to Tomcat 10
2.3 Make the integration-tests-engine-jakarta module test build functional with Tomcat 10
  • 2.3.1 Add a resources-tomcat directory with an arquillian.xml, similar to the one that tomcat 9 uses and make it depend on the Tomcat 10 property tomcat.runtime.location
  • 2.3.1 Add the CDI Implementation test dependency to the pom.xml org.jboss.weld.servlet:weld-servlet-shaded:4.0.3.Final
  • 2.3.2 Modify the maven-resources-plugin
    • Add an execution called copy-test-sources-tomcat which copies the content of ${basedir}/../integration-tests-engine/src/test/java-tomcat into ${project.build.directory}/generated-test-sources/java-tomcat
    • Add an execution called copy-test-resources-tomcat which copies the content of >${basedir}/../integration-tests-engine/src/test/resources-tomcat into ${project.build.directory}/generated-test-resources/resources-tomcat and excludes the arquillian.xml and uses the Tomcat 10 arquillian.xml, found in resources-tomcat instead
  • 2.3.2 Modify the transformer-maven-plugin
    • Add an execution called transform-jakarta-test-sources-tomcat which will transform the ${project.build.directory}/generated-test-sources/java-tomcat resources to the jakarta namespace.
    • Add an execution called transform-jakarta-test-resources-tomcat which will tranform the ${project.build.directory}/generated-test-resources/resources-tomcat resources into the jakarta namespace
  • 2.3.3 Add a Tomcat profile for building and executing the tests
    • Add the following dependencies to the profile
      • org.camunda.bpm.qa:camunda-qa-tomcat-runtime:${project.version}:pom in provided scope
      • org.jboss.arquillian.container:arquillian-tomcat-managed-8 in test scope
      • org.jboss.arquillian.protocol:arquillian-protocol-servlet-jakarta which will use the Jakarta Servlet 5.0 arquillian protocol
      • org.jboss.arquillian.junit:arquillian-junit-container in test scope
    • Add a build section to the profile which should do the following things in order to run the tests
      • Use src/test/resources-tomcat and resources-tomcat as test resources
      • Use the build-helper-maven-plugin to add java-tomcat as a test source
      • Use the maven-surefire-plugin to execute the unit tests against Tomcat 10
2.4 Make the camunda-external-task-client use the latest Tomcat 10 container against tests
  • Use the jakarta flavor of the following artifacts: camunda-tomcat-assembly, camunda-engine-rest
  • Replace the maven-dependency-plugin artifact used with the jakarta version: camunda-tomcat-assembly-jakarta
  • Make the cargo-maven3-plugin to use Tomcat 10; Rename the containerId used to tomcat10x
  • Make the deployable use the jakarta flavor camunda-engine-rest-jakarta instead
  • Make the module engine-variable-test use the artifactjakarta.servlet:jakarta.servlet-api instead
2.5 Change integration-tests-webapps/shared-engine to support both Tomcat 9 and 10 profiles
  • Change the tomcat profile to tomcat9; make it depend on the artifact camunda-qa-tomcat9-runtime
  • Add a new profile for tomcat 10, similar to tomcat9 profile using the jakarta artifacts and the tomcat 10 properties. Dependencies include camunda-qa-tomcat-runtime, camunda-webapp-tomcat-jakarta, camunda-engine-rest-jakarta, similar to the tomcat9 profile using the equivalent jakarta flavors
  • Add a skipTests entry to the maven-surefire-plugin of the qa module which is controlled from a property e.g skipTests-webapps
    • camunda-qa module contains the profile jdk17 which is activated for Java 17, thus Tomcat 10. This profile includes the integration-tests-engine-jakarta which will run the surefire tests. In order to exclude the tests for the webapps-integration profile, this property can be used to skip the tests

3. CI
3.1 Adjust the CamBPM Main Pipeline to use the Tomcat 9 dependencies
  • Adjust engine-IT-tomcat-9-postgresql-142 and webapp-IT-tomcat-9-h2 stages to use the tomcat9 profile instead
3.2 Add the new Tomcat 10 stages to the Main Pipeline
  • Copy the tomcat 9 stages and use them as a basis for tomcat 10 stages; rename the stages to have the tomcat suffix (tomcat suffix conventionally implies the latest version (similarly to wildfly))
  • Add a new stage for engine-IT tests and one for the webapps
    • Stage: engine-IT-tomcat-10-postgresql-142; Maven command: clean install -Ptomcat,postgresql,engine-integration-jakarta
    • Stage: webapp-IT-tomcat-10-h2; Maven command: clean install -Ptomcat,h2,webapps-integration -DskipTests-webapps=true'
3.3 Adjust the CamBPM EE Main Pipeline to use the Tomcat 9 dependencies
  • Adjust EE-webapp-IT-tomcat-h2 stage to use the tomcat9 profile instead
3.4 Add the new Tomcat 10 stage to the EE Main Pipeline
  • Add a new stage for webapps
    • Stage: webapp-IT-tomcat-10-h2; Maven command: clean install -Ptomcat,h2,webapps-integration -DskipTests-webapps=true'

4. Docker Image

The Docker image of 7.22 will use the latest Tomcat 10 version.

5. camunda-bpm-platform-ee

Similar to the CE, the EE repo will support only Tomcat 10

Changes required for Tomcat 10 Support

  • 5.1 tomcat/distro module needs to use the latest tomcat 10 artifacts and properties
    • assembly.xml needs to use the jakarta artifacts for engine-rest, camunda-example-invoice, camunda-tomcat-assembly-jakarta
    • Use the property version.tomcat which points to latest Tomcat 10 version
  • 5.2 camunda/qa-ee module needs to execute tests against tomcat 10
    • Update the tomcat profile of tomcat-runtime to utilise the latest tomcat 10 dependencies and properties
      • Use camunda-tomcat-assembly-jakarta in the pom.xml instead
      • Use the latest version.tomcat to point to latest tomcat 10
  • 5.3 The camunda-webapp-ee-plugins need to include the Tomcat 10 dependencies instead of Tomcat 9
    • camunda-webapp-jakarta needs to be used instead of camunda-webapp
    • jboss-javaee-6.0 runtime needs to be replaced with the weld-shaded-servlet uber jar
  • 5.4 The CI pipeline EE-webapp-IT-tomcat-h2 needs to be adapted to use the Tomcat 10 artifacts

6. Documentation

6.1 Changes Required in camunda-docs-manual
6.1.1 Add Tomcat to Update & Migration Section
6.1.2 Update the Supported Environments
  • Update the Tomcat version to include both 9 & 10
6.1.3 Update the Job Executor Configuration Section
  • Update the Tomcat reference which currently points to Tomcat 9 to point to the latest Tomcat 10 version.
6.1.4 Update the Tomcat Manual Installation Section

Similar to the Wildfly Manual Installation Guide, the Tomcat 10 Manual Installation Guide needs to be updated to include steps for both Tomcat 9 and Tomcat 10 versions:

  • 6.1.4.1 The REST API - Step 1 needs to split into two sections, for Tomcat 9 to mention camunda-engine-rest and for Tomcat 10 the camunda-engine-rest-jakarta wars respectively. Here is the respective Wildfly section
  • 6.1.4.2 The Secure Tomcat section needs to point the Security HowTo Guide for both Tomcat 9 and Tomcat 10
  • 6.1.4.3 The Cockpit, Tasklist and Admin section needs to mention both camunda-webapp-tomcat and camunda-webapp-tomcat-jakarta artifacts both for Tomcat 9 and Tomcat 10 respectively. Similar case for Wildfly can be found here

7. Backport - Tomcat 10 to 7.21

In order to provide support for Tomcat 10 for the 7.21 backport, we can provide all the necessary module-building blocks and allow our users to build it themselves.

  • We don't provide a dedicated Tomcat 10 distro out of the box for our users
  • Docker Support is constrained by Tomcat 9 distro ; no Tomcat 10 Docker image will be provided to our users. If they want they can build it themselves
  • The CI will execute the engine-integration-tests-jakarta and integration-tests-webapps against Tomcat 10 to ensure coverage
  • In the EE pipeline, the integration-tests-webapps will be executed only against Tomcat 9

8. License Check

Tomcat 10 is licensed under Apache 2.0 which falls under the Go list; no license check is required for the server.

The dependencies of the Pull Request are still subjects to the usual license check.

Breakdown

Work can be split into two main tickets: one for the feature and another one for the backport. Each ticket can encapsulate the changes to the main repo, the ee-repo, docker and documentation.

The changes might include numerous files in the main repo. Since the changes are functionally cohesive (they change together), it makes sense to review them together in one review. Below you can find one high level task that can be implemented, tested and reviewed individually:

  • Add Tomcat 10 Support to 7.22

    • [ ] Add Tomcat 10 Support to codebase camunda-bpm-platform
    • [ ] Add Tomcat 10 Support to codebase camunda-bpm-platform-ee
    • [ ] Add Tomcat 10 Support to Docker image
    • [ ] Add Documentation

  • Backport Tomcat 10 Support to 7.21

    • [ ] Add Tomcat 10 Support to codebase camunda-bpm-platform
    • [ ] Add Tomcat 10 Support to codebase camunda-bpm-platform-ee
    • [ ] Add Tomcat 10 Support to Docker image
    • [ ] Add Documentation

  • [ ] Restore Tomcat 10 Failing Tests for 7.21

  • [ ] Restore Tomcat 10 Failing Tests for 7.22

Note: Here is the relevant investigation thread for the failing integration-tests against Tomcat 10

Glossary

❓ Open point for Decision

psavidis avatar Apr 22 '24 15:04 psavidis

Progress so far with Tomcat 10 ✔

Engine Integration Tests Problems

  • Problem: Currently, the engine-integration tests start tomcat 10 and then there are failures in the tomcat logs
  • Context:

    • Weld and CDI need to be picked up from engine-cdi-jakarta's dependency management

    • org.jboss.weld.servlet:weld-servlet-shaded:4.0.3.Final might be causing a conflict (bundles CDI and weld)

      • It might need to be replaced with another non shaded jar
      • A similar jakartified dependency is needed, similar to jboss-javaee-6.0-with-tools
  • Answer: here

Webapp Integration Test Problems

  • Problem: When Tomcat 9 is used as a profile to start the webapp-integration tests, the log output of cargo-maven3-plugin in the shared-engine module outputs logs that are specific to the boostraping of Tomcat AND logs of REASTEASY runtime when it is initialised propertly. For Tomcat 10, only the logs of reasteasy are appearing and tomcat is missing
  • Context: The bpm-platform.xml is correctly copied in the Tomcat 10 runtime of QA tests and is not picked up. If it would be, the TomcatParseBpmPlatformXmlStep class would ouput its logs along with other engine specific logs.
  • Hypothesis: The engine is not getting initialised propertly.
  • Answer: here

psavidis avatar May 17 '24 13:05 psavidis

Latest Investigation findings

  • Tomcat 10 Webapps Latest Investigation

  • After debugging cargo-maven3-plugin and the Tomcat 10 startup, we were able to set a breakpoint at TomcatBpmPlatformBootstrap and see that on StartEvent, the bpm-platform.xml is loaded correctly. However, the logs are not visible!

  • The invoice application also is started (the flow stops at a breakpoint during debugging)

  • The engine-rest does not work for distro

Next steps:

  1. Check engine-rest of tomcat distro
  2. There is something wrong with the logging
  3. Explore what happens with the invoice application and engine-rest and see what happens there

psavidis avatar May 17 '24 14:05 psavidis

Investigation Update ✔

Debugging

  • Calling Tomcat 9 at http://localhost:8080/engine-rest/

    • Breaks during debugging (CacheControlFilter#doFilter)
    • Throws a NotFoundException
    • Results in a JSON response

  • Calling Tomcat 10 at http://localhost:8080/engine-rest/

    • Breaks during debugging (CacheControlFilter#doFilter)
    • Returns The Tomcat 404 page

Problem: ExceptionHandler is not invoked for Tomcat 10.DefaultApplication is being registered and all handlers are registered but not invoked. Root-cause: The engine-rest/assembly-jakarta/src/main/runtime/tomcat/webapp/WEB-INF/web.xml descriptor which registers the Reaseasy HttpServletDispatcher initialises the parameter for Application. The name currently is javax.ws.rs.Application which is wrong for Jakarta servlets. Solution: Change the name to jakarta.ws.rs.Application

psavidis avatar May 20 '24 08:05 psavidis

Investigation Update ✔

The above change fixed the webapp-integration tests against Tomcat 10

psavidis avatar May 20 '24 09:05 psavidis

Investigation Update ✔

  • DeploymentHelperresolves programmatically from pom.xml runtime dependencies; for the case of Tomcat 10 and jakarta, this renders the necessity for weld-servlet-core to 1. exist in the pom.xml with a concrete version; this means the version cannot be inherited from camunda-engine-cdi-jakarta module
  • With the Latest changes, the engine-integration-jakarta profile throws a Caused by: java.lang.ClassNotFoundException: jakarta.inject.Provider
    • Tried to add the dependencies jakarta.inject:jakarta.inject-api, jakarta.enterprise:jakarta.enterprise.cdi-api in compile, test scope and the same error is thrown

Solution: Removed weld-servlet-shaded and used instead the following dependencies:weld-servlet-core, jakarta.inject-api, jakarta.transaction-api, jakarta.enterprise.concurrent-api; also, used the DeploymentHelper class to put these dependencies in the wars the tests create and the exceptions were removed from the logs.

psavidis avatar May 20 '24 13:05 psavidis

Investigation Update

  • Problem 1 - The webapp-integration profile executes the engine integration tests; it shouldn't.
    • Reason: Maven reactor adds the QA Integration Tests - Engine - Jakarta module to the execution for Tomcat 10 whereas this module is missing from execution for tomcat 9; here are the maven command executions by the webapp integration stage:
      • Tomcat 9: mvn -s $MAVEN_SETTINGS_XML clean install -Ptomcat9,h2,webapps-integration -nsu -Dmaven.repo.local=${WORKSPACE}/.m2 -f qa//pom.xml -B
      • Tomcat 10: mvn -s $MAVEN_SETTINGS_XML clean install -Ptomcat,h2,webapps-integration -nsu -Dmaven.repo.local=${WORKSPACE}/.m2 -f qa//pom.xml -B
      • The addition of integration-tests module seems to be added to the reactor only for java17 and not for java11
      • camunda-qa has a jdk17 profile which is triggered on java 17 usage and adds the integration-tests-engine-jakarta module to the execution; it should be skipped for the above execution.
    • Solution: Add a flag skipTests-webapps to conditionally skip the surefire integration tests for the webapps-integration profile and tomcat; Use that at the newly introduced stage.
  • Problem 2 - Logging Output of catalina.out on Tomcat 10
    • The logging of ContainerIntegrationLogger#camundaBpmPlatformSuccessfullyStarted is not displayed in the logs
    • Also, the logs of spin are not found in catalina.out
  • Problem 3 - Tomcat9 Ejb services are initialised for TestDeploymentSource but not for Tomcat10
    • DeployProcessArchiveStep#getProcessEngine:223 tries to fetch the default process engine bean and it cannot.
    • Reason: The RuntimeContainerDelegateImpl is instantiated 1 time for Tomcat 9 and Java 11 and 2 times for Tomcat 10 and Java 17; The second time, the container that is used by the JakartaServletProcessApplication is the second which is empty!
      • See Investigation reference here

psavidis avatar May 22 '24 11:05 psavidis

Investigation Update

Tomcat 10

  • TomcatBpmPlatformBootstrap calls RuntimeContainerDelegate.INSTANCE.get() -> Creates a new RuntimeContainerDelegateImpl
  • JakartaServletProcessApplication#deploy calls RuntimeContainerDelegate.INSTANCE.get() -> Creates a new RuntimeContainerDelegateImpl instead of reusing the same one

Tomcat 9

  • TomcatBpmPlatformBootstrap calls RuntimeContainerDelegate.INSTANCE.get() -> Creates a RuntimeContainerDelegateImpl
  • ServletProcessApplication#deploy calls `RuntimeContainerDelegate.INSTANCE.get() but does not create a new instance

Assumption: RuntimeContainerDelegate has a static inner class which creates a RuntimeContainerDelegateImpl only upon access (relying on the JVM's lifecycle for creating instance fields of static classes).

  • Tomcat 9 is accessing it correctly from the classpath and creates it only once.
  • Tomcat 10 must have a classloading issue which loads the RuntimeContainerDelegate.INSTANCE a second time and it results into a new creation
  • Hint: The above classes belong to the engine which for the case of Tomcat 10 exists in the lib folder
  • Hint2:
    • On Tomcat 9, RuntimeContainerDelegateImpl is loaded by Java's classloader java.net.URLClassLoader
    • On Tomcat 10, RuntimeContainerDelegateImpl is loaded twice, first by java.net.URLClassLoader and a second time by ParallelWebappClassLoader
      • ParallelWebappClassLoader context: test delegate: false ----------> Parent Classloader: java.net.URLClassLoader@4f063c0a

psavidis avatar May 23 '24 14:05 psavidis

Investigation Update ✔

The above issue happens only upon debugging. When executing the integration-tests using the following command:

mvn clean install -Ptomcat,h2,engine-integration-jakarta

The ouput is the following:

[INFO] Running org.camunda.bpm.integrationtest.service.ProcessEngineServiceTest [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 127.188 s <<< FAILURE! - in org.camunda.bpm.integrationtest.service.ProcessEngineServiceTest [ERROR] org.camunda.bpm.integrationtest.service.ProcessEngineServiceTest Time elapsed: 127.184 s <<< ERROR! org.jboss.arquillian.container.spi.client.container.LifecycleException: Could not start container Caused by: java.util.concurrent.TimeoutException: Managed server was not started within [120] s

Tomcat does not start.

Why: The issue was caused by arquillian.xml when debugging is enabled; After arquillian is started, if a remote debugger is not connected, arquillian waits the max timeout and then throws the TimeoutException Solution: Disable arquillian debugging

psavidis avatar May 24 '24 08:05 psavidis

Investigation Update

  • The code with weld-servlet-shaded and the changes of this PR makes many integration-tests pass and the failures look less on the CI
  • Locally the above code throws TimeoutException, both on the new branch and tomcat10_support_latest_master; there must be a discrepancy on the local environment which is unidentified yet.

psavidis avatar May 24 '24 12:05 psavidis

Investigation Update

Context: : TimerRecalculationTest fails on Tomcat 10,h2 Test: TimerRecalculationTest#testTimerRecalculationBasedOnProcessVariable Build Command: mvn clean install -Ptomcat,h2,engine-integration-jakarta -DfailIfNoTests=false -Dtest=TimerRecalculationTest Problem: When the JUEL expression is evaluated atTimeDeclarationImpl Line 129 the above exception is thrown

  • Prior to the test execution test1.war is deployed to Tomcat 10 and inside the war, the missing class org.jboss.weld.proxy.WeldConstruct is there.

Why: The issue is related to an Arquillian Regression issue; weld-servlet-shaded is included inside test1.war. When the same jar is put manually in the lib folder of Tomcat, the test passes.

  • Issue ARQ-2234 was created to track the problem
  • Kudos to @tasso94 for root-causing the issue
  • Investigation Link can be found here
Exception Stack Error while evaluating expression: ${timerExpressionBean.getTimerDuration()}. Cause: WELD-001524: Unable to load proxy class for bean Producer Method [ProcessEngine] with qualifiers [@Default @Any @Named] declared as [[BackedAnnotatedMethod] @Produces @Named @ApplicationScoped public org.camunda.bpm.engine.cdi.impl.ProcessEngineServicesProducer.processEngine()] with class class java.lang.Object

Caused By: java.lang.NoClassDefFoundError: org/jboss/weld/proxy/WeldConstruct

Tried the following:

  • Upgraded to org.jboss.weld.servlet:weld-servlet-shaded:5.1.0.Final; version 5.x is required to work with Tomcat 10.1.18 according to this source
  • Put the missing weld-api in the war using the DeploymentHelper, similarly to how weld-servlet-shaded
  • Tried including the weld-api in the pom.xml
  • Tried including the artifacts of the shaded plugin one by one in the pom.xml
  • Tried excluding the weld-api from camunda-engine-cdi-jakarta in case of a version conflict

Update 5/05/2024: Tried to investigate the possibility there is a reflection conflict between the weld-servlet-shaded library and some other, like this similar thread

  • Observation: When weld-cdi is moved out of the WEB-INF/lib folder of the exploded war and moved into the lib folder of Tomcat, the problem is resolved.
  • Description:
    • Unzipped all Tomcat jars and used a script to search for the WeldConstruct.class; it is not found two times.
    • Tried excluding org.jboss.weld artifacts from engine-cdi-jakarta

psavidis avatar May 27 '24 14:05 psavidis

Investigation Update ✔

Problem: Slf4jClassloadingTest Fails Why: shouldNotUseNopLoggerFactory fails with an assertion if Slf4j is not found on the classpath Solution: Add slf4j-jdk14 on the classpath of qa/tomcat-runtime

psavidis avatar May 29 '24 06:05 psavidis

I raised https://issues.redhat.com/browse/ARQ-2234 in the Arquillian issue tracker to investigate the following exception when accessing beans in Camunda 7:

Exception stack trace 
-------------------------------------------------------------------------------
Test set: org.jboss.arquillian.container.tomcat.managed.bean.AccessBeanIT
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 9.817 s <<< FAILURE! -- in org.jboss.arquillian.container.tomcat.managed.bean.AccessBeanIT
org.jboss.arquillian.container.tomcat.managed.bean.AccessBeanIT.shouldAccessVariableWithoutException -- Time elapsed: 0.195 s <<< ERROR!
org.camunda.bpm.engine.ProcessEngineException: Error while evaluating expression: ${timerExpressionBean.getTimerDuration()}. Cause: WELD-001524: Unable to load proxy class for bean Producer Method [ProcessEngine] with qualifiers [@Default @Any @Named] declared as [[BackedAnnotatedMethod] @Produces @Named @ApplicationScoped public org.camunda.bpm.engine.cdi.impl.ProcessEngineServicesProducer.processEngine()] with class class java.lang.Object
	at org.camunda.bpm.engine.impl.el.JuelExpression.getValue(JuelExpression.java:76)
	at org.camunda.bpm.engine.impl.el.JuelExpression.getValue(JuelExpression.java:51)
	at org.camunda.bpm.engine.impl.jobexecutor.TimerDeclarationImpl.resolveAndSetDuedate(TimerDeclarationImpl.java:129)
	at org.camunda.bpm.engine.impl.jobexecutor.TimerDeclarationImpl.initializeConfiguration(TimerDeclarationImpl.java:97)
	at org.camunda.bpm.engine.impl.jobexecutor.TimerDeclarationImpl.postInitialize(TimerDeclarationImpl.java:156)
	at org.camunda.bpm.engine.impl.jobexecutor.TimerDeclarationImpl.postInitialize(TimerDeclarationImpl.java:38)
	at org.camunda.bpm.engine.impl.jobexecutor.JobDeclaration.createJobInstance(JobDeclaration.java:127)
	at org.camunda.bpm.engine.impl.jobexecutor.TimerDeclarationImpl.createTimer(TimerDeclarationImpl.java:182)
	at org.camunda.bpm.engine.impl.jobexecutor.TimerDeclarationImpl.createTimerInstance(TimerDeclarationImpl.java:167)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.createTimerInstances(ExecutionEntity.java:418)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.initializeTimerDeclarations(ExecutionEntity.java:410)
	at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl.enterActivityInstance(PvmExecutionImpl.java:1317)
	at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationActivityInstanceStart.eventNotificationsStarted(PvmAtomicOperationActivityInstanceStart.java:41)
	at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationActivityInstanceStart.eventNotificationsStarted(PvmAtomicOperationActivityInstanceStart.java:35)
	at org.camunda.bpm.engine.impl.core.operation.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:47)
	at org.camunda.bpm.engine.impl.interceptor.AtomicOperationInvocation.execute(AtomicOperationInvocation.java:99)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.invokeNext(CommandInvocationContext.java:141)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performNext(CommandInvocationContext.java:121)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:96)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:622)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:596)
	at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationTransitionCreateScope.scopeCreated(PvmAtomicOperationTransitionCreateScope.java:38)
	at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationCreateScope.execute(PvmAtomicOperationCreateScope.java:53)
	at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationCreateScope.execute(PvmAtomicOperationCreateScope.java:27)
	at org.camunda.bpm.engine.impl.interceptor.AtomicOperationInvocation.execute(AtomicOperationInvocation.java:99)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.invokeNext(CommandInvocationContext.java:141)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performNext(CommandInvocationContext.java:128)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext$1.call(CommandInvocationContext.java:112)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext$1.call(CommandInvocationContext.java:110)
	at org.camunda.bpm.engine.impl.context.ProcessApplicationClassloaderInterceptor.call(ProcessApplicationClassloaderInterceptor.java:48)
	at org.camunda.bpm.application.AbstractProcessApplication.execute(AbstractProcessApplication.java:131)
	at org.camunda.bpm.application.AbstractProcessApplication.execute(AbstractProcessApplication.java:142)
	at org.camunda.bpm.engine.impl.context.Context.executeWithinProcessApplication(Context.java:208)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performNext(CommandInvocationContext.java:110)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:96)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:622)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:596)
	at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl.start(PvmExecutionImpl.java:294)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.start(ExecutionEntity.java:449)
	at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl.start(PvmExecutionImpl.java:271)
	at org.camunda.bpm.engine.impl.cmd.StartProcessInstanceCmd.execute(StartProcessInstanceCmd.java:66)
	at org.camunda.bpm.engine.impl.cmd.StartProcessInstanceCmd.execute(StartProcessInstanceCmd.java:38)
	at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:28)
	at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:111)
	at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:70)
	at org.camunda.bpm.engine.impl.interceptor.CommandCounterInterceptor.execute(CommandCounterInterceptor.java:34)
	at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:33)
	at org.camunda.bpm.engine.impl.interceptor.ExceptionCodeInterceptor.execute(ExceptionCodeInterceptor.java:55)
	at org.camunda.bpm.engine.impl.ProcessInstantiationBuilderImpl.executeWithVariablesInReturn(ProcessInstantiationBuilderImpl.java:166)
	at org.camunda.bpm.engine.impl.ProcessInstantiationBuilderImpl.execute(ProcessInstantiationBuilderImpl.java:132)
	at org.camunda.bpm.engine.impl.ProcessInstantiationBuilderImpl.execute(ProcessInstantiationBuilderImpl.java:128)
	at org.camunda.bpm.engine.impl.RuntimeServiceImpl.startProcessInstanceByKey(RuntimeServiceImpl.java:88)
	at org.jboss.arquillian.container.tomcat.managed.bean.AccessBeanIT.shouldAccessVariableWithoutException(AccessBeanIT.java:77)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.jboss.arquillian.junit.Arquillian$8$1.invokeMethod(Arquillian.java:319)
	at org.jboss.arquillian.junit.MethodInvoker$1.invoke(MethodInvoker.java:18)
	at org.jboss.arquillian.container.test.impl.execution.LocalTestExecuter.execute(LocalTestExecuter.java:57)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
	at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103)
	at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:90)
	at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:133)
	at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:105)
	at org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:62)
	at org.jboss.arquillian.container.test.impl.execution.ContainerTestExecuter.execute(ContainerTestExecuter.java:36)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
	at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103)
	at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:90)
	at org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:116)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
	at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:95)
	at org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:83)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
	at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:95)
	at org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:69)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
	at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:95)
	at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:133)
	at org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.test(EventTestRunnerAdaptor.java:139)
	at org.jboss.arquillian.junit.MethodInvoker.invoke(MethodInvoker.java:15)
	at org.jboss.arquillian.junit.Arquillian$8.evaluate(Arquillian.java:326)
	at org.jboss.arquillian.junit.Arquillian$4.evaluate(Arquillian.java:198)
	at org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:344)
	at org.jboss.arquillian.junit.Arquillian.access$200(Arquillian.java:48)
	at org.jboss.arquillian.junit.Arquillian$5.evaluate(Arquillian.java:209)
	at org.jboss.arquillian.junit.Arquillian$7$1.invoke(Arquillian.java:273)
	at org.jboss.arquillian.container.test.impl.execution.BeforeLifecycleEventExecuter.on(BeforeLifecycleEventExecuter.java:34)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
	at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103)
	at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:90)
	at org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:116)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
	at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:95)
	at org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:83)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
	at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:95)
	at org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:69)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
	at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:95)
	at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:133)
	at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:105)
	at org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.fireCustomLifecycle(EventTestRunnerAdaptor.java:159)
	at org.jboss.arquillian.junit.Arquillian$7.evaluate(Arquillian.java:267)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.jboss.arquillian.junit.Arquillian$2.evaluate(Arquillian.java:160)
	at org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:344)
	at org.jboss.arquillian.junit.Arquillian.access$200(Arquillian.java:48)
	at org.jboss.arquillian.junit.Arquillian$3.evaluate(Arquillian.java:171)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.jboss.arquillian.junit.Arquillian.run(Arquillian.java:109)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at org.jboss.arquillian.junit.container.JUnitTestRunner.execute(JUnitTestRunner.java:61)
	at org.jboss.arquillian.protocol.servlet5.runner.ServletTestRunner.executeTest(ServletTestRunner.java:139)
	at org.jboss.arquillian.protocol.servlet5.runner.ServletTestRunner.execute(ServletTestRunner.java:117)
	at org.jboss.arquillian.protocol.servlet5.runner.ServletTestRunner.doGet(ServletTestRunner.java:86)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:564)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:663)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:389)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:896)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.jboss.weld.exceptions.WeldException: WELD-001524: Unable to load proxy class for bean Producer Method [ProcessEngine] with qualifiers [@Default @Any @Named] declared as [[BackedAnnotatedMethod] @Produces @Named @ApplicationScoped public org.camunda.bpm.engine.cdi.impl.ProcessEngineServicesProducer.processEngine()] with class class java.lang.Object
	at org.jboss.weld.bean.proxy.ProxyFactory.getProxyClass(ProxyFactory.java:421)
	at org.jboss.weld.bean.proxy.ProxyFactory.run(ProxyFactory.java:372)
	at org.jboss.weld.bean.proxy.ProxyFactory.create(ProxyFactory.java:364)
	at org.jboss.weld.bean.proxy.ClientProxyFactory.create(ClientProxyFactory.java:83)
	at org.jboss.weld.bean.proxy.ClientProxyProvider.createClientProxy(ClientProxyProvider.java:207)
	at org.jboss.weld.bean.proxy.ClientProxyProvider.createClientProxy(ClientProxyProvider.java:197)
	at org.jboss.weld.bean.proxy.ClientProxyProvider$CreateClientProxy.apply(ClientProxyProvider.java:52)
	at org.jboss.weld.bean.proxy.ClientProxyProvider$CreateClientProxy.apply(ClientProxyProvider.java:48)
	at org.jboss.weld.util.cache.ReentrantMapBackedComputingCache.lambda$new$0(ReentrantMapBackedComputingCache.java:57)
	at org.jboss.weld.util.LazyValueHolder$1.computeValue(LazyValueHolder.java:32)
	at org.jboss.weld.util.LazyValueHolder.get(LazyValueHolder.java:46)
	at org.jboss.weld.util.cache.ReentrantMapBackedComputingCache.getValue(ReentrantMapBackedComputingCache.java:74)
	at org.jboss.weld.util.cache.ReentrantMapBackedComputingCache.getCastValue(ReentrantMapBackedComputingCache.java:80)
	at org.jboss.weld.bean.proxy.ClientProxyProvider.getClientProxy(ClientProxyProvider.java:232)
	at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:703)
	at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:815)
	at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:93)
	at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:348)
	at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:359)
	at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70)
	at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:49)
	at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72)
	at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:126)
	at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:165)
	at org.jboss.weld.contexts.unbound.DependentContextImpl.get(DependentContextImpl.java:64)
	at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:104)
	at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
	at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:706)
	at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:813)
	at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:93)
	at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:348)
	at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:359)
	at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70)
	at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:49)
	at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72)
	at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:126)
	at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:165)
	at org.jboss.weld.contexts.unbound.DependentContextImpl.get(DependentContextImpl.java:64)
	at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:104)
	at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
	at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:706)
	at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:813)
	at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:93)
	at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:348)
	at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:359)
	at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70)
	at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:49)
	at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72)
	at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:126)
	at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:165)
	at org.jboss.weld.contexts.unbound.DependentContextImpl.get(DependentContextImpl.java:64)
	at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:104)
	at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
	at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:706)
	at org.jboss.weld.injection.producer.AbstractMemberProducer.getReceiver(AbstractMemberProducer.java:128)
	at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:164)
	at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:187)
	at org.jboss.weld.contexts.unbound.DependentContextImpl.get(DependentContextImpl.java:64)
	at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:104)
	at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
	at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:706)
	at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:813)
	at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:93)
	at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:348)
	at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:359)
	at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70)
	at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:49)
	at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72)
	at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:126)
	at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:165)
	at org.jboss.weld.contexts.unbound.DependentContextImpl.get(DependentContextImpl.java:64)
	at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:104)
	at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
	at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:706)
	at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:729)
	at org.camunda.bpm.engine.cdi.impl.util.ProgrammaticBeanLookup.getContextualReference(ProgrammaticBeanLookup.java:118)
	at org.camunda.bpm.engine.cdi.impl.util.ProgrammaticBeanLookup.lookup(ProgrammaticBeanLookup.java:65)
	at org.camunda.bpm.engine.cdi.impl.util.ProgrammaticBeanLookup.lookup(ProgrammaticBeanLookup.java:54)
	at org.camunda.bpm.engine.cdi.impl.el.CdiResolver.getValue(CdiResolver.java:65)
	at org.camunda.bpm.impl.juel.jakarta.el.CompositeELResolver.getValue(CompositeELResolver.java:136)
	at org.camunda.bpm.engine.impl.el.AbstractElResolverDelegate.getValue(AbstractElResolverDelegate.java:69)
	at org.camunda.bpm.impl.juel.jakarta.el.CompositeELResolver.getValue(CompositeELResolver.java:136)
	at org.camunda.bpm.impl.juel.AstIdentifier.eval(AstIdentifier.java:81)
	at org.camunda.bpm.impl.juel.AstMethod.invoke(AstMethod.java:79)
	at org.camunda.bpm.impl.juel.AstMethod.eval(AstMethod.java:75)
	at org.camunda.bpm.impl.juel.AstEval.eval(AstEval.java:50)
	at org.camunda.bpm.impl.juel.AstNode.getValue(AstNode.java:26)
	at org.camunda.bpm.impl.juel.TreeValueExpression.getValue(TreeValueExpression.java:112)
	at org.camunda.bpm.engine.impl.delegate.ExpressionGetInvocation.invoke(ExpressionGetInvocation.java:40)
	at org.camunda.bpm.engine.impl.delegate.DelegateInvocation.proceed(DelegateInvocation.java:58)
	at org.camunda.bpm.engine.impl.delegate.DefaultDelegateInterceptor.handleInvocationInContext(DefaultDelegateInterceptor.java:92)
	at org.camunda.bpm.engine.impl.delegate.DefaultDelegateInterceptor.handleInvocation(DefaultDelegateInterceptor.java:63)
	at org.camunda.bpm.engine.impl.el.JuelExpression.getValue(JuelExpression.java:60)
	... 189 more
Caused by: java.lang.NoClassDefFoundError: org/jboss/weld/proxy/WeldConstruct
	at java.base/java.lang.ClassLoader.defineClass0(Native Method)
	at java.base/java.lang.System$2.defineClass(System.java:2307)
	at java.base/java.lang.invoke.MethodHandles$Lookup$ClassDefiner.defineClass(MethodHandles.java:2439)
	at java.base/java.lang.invoke.MethodHandles$Lookup$ClassDefiner.defineClass(MethodHandles.java:2416)
	at java.base/java.lang.invoke.MethodHandles$Lookup.defineClass(MethodHandles.java:1843)
	at org.jboss.weld.bean.proxy.util.WeldDefaultProxyServices.defineWithMethodLookup(WeldDefaultProxyServices.java:153)
	at org.jboss.weld.bean.proxy.util.WeldDefaultProxyServices.defineClass(WeldDefaultProxyServices.java:62)
	at org.jboss.weld.bean.proxy.ProxyFactory.toClass(ProxyFactory.java:968)
	at org.jboss.weld.bean.proxy.ProxyFactory.createProxyClass(ProxyFactory.java:509)
	at org.jboss.weld.bean.proxy.ProxyFactory.getProxyClass(ProxyFactory.java:413)
	... 281 more
Caused by: java.lang.ClassNotFoundException: org.jboss.weld.proxy.WeldConstruct
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:587)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
	... 291 more

I created the following minimal reproducible example: https://github.com/tasso94/arquillian-container-tomcat/commit/fedc88

To run it, clone the repo, checkout branch tomcat-10-test and run mvn clean install. The test org.jboss.arquillian.container.tomcat.managed.bean.AccessBeanIT should fail.

tasso94 avatar May 29 '24 08:05 tasso94

Investigation Update

Remaining failing tests assessment:

  • Test: FailingJobBoundaryTimerWithDelegateVariablesTest
    • Symptom: Fails with NullPointerException
    • Why: RuntimeService & ManagementService are not injected by the @Inject annotations after deployment and when the test starts. The services in the AbstractFoxPlatformIntegrationTest are initialised.
    • Observation: For Tomcat 9, the RuntimeService is injected with a Weld Proxy e.g with a bean name such as RuntimeService$1712672648$Proxy$_$$_WeldClientProxy@9864.
    • Assumption: It's reasonable to assume Dependency Injection does not work due to unitiiliased Weld; The test might pass when the classloading issue of WeldConstruct is resolved.
  • Test: CallActivityContextSwitchTest#testNoWaitState
    • Symptom: Fails with NullPointerException
    • Why: Same issue as above
  • Test: SignalEventCatchBoundaryWithVariablesTest
    • Symptom: Fails with NullPointerException
    • Why: Same issue as above
  • Test: CallActivityContextSwitchTest.testMainASyncBeforeCalledASync
    • Symptom: Fails with assertion error (expected:<true> but was:<null>)
    • Why: The assertion is related to the missing class org.jboss.weld.proxy.WeldConstruct
    • Solution: Most likely, it will be fixed when the classloading of WeldConstruct is restored

The rest of CallActivityContextSwitchTest all have a relation to the classloading issue of WeldConstruct

psavidis avatar May 29 '24 10:05 psavidis

Decision Thread Update

The Backport to 7.21 has been approved by Product management.

psavidis avatar Jun 17 '24 13:06 psavidis

Update

After merging the last PR for updating 7.21 docs, the ticket will be assigned to the QA for manual testing.

psavidis avatar Aug 13 '24 12:08 psavidis

Update

Assigning the ticket to the QA for manual testing.

psavidis avatar Aug 14 '24 07:08 psavidis

Hi @psavidis, I changed the tag potential:7.21.4 to version:7.21.4.

tasso94 avatar Aug 19 '24 09:08 tasso94