ignite icon indicating copy to clipboard operation
ignite copied to clipboard
verified publisher icon apache

Apache Ignite 2.17.0 Fails to Start with OPTION_LIBS=ignite-rest-http Due to Missing Jackson Dependency

Open gammacomputer opened this issue 10 months ago • 9 comments

Description:

When starting Apache Ignite 2.17.0 with OPTION_LIBS=ignite-rest-http, the process fails with a java.lang.NoClassDefFoundError for com.fasterxml.jackson.core.JsonProcessingException. The problem does not occur in version 2.16.0.

Error Message:

[11:17:48,997][SEVERE][main][IgniteKernal] Exception during start processors, node will be stopped and close connections
java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/JsonProcessingException
        at org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestProtocol.start(GridJettyRestProtocol.java:104)
        at org.apache.ignite.internal.processors.rest.GridRestProcessor.startProtocol(GridRestProcessor.java:1037)
        at org.apache.ignite.internal.processors.rest.GridRestProcessor.startHttpProtocol(GridRestProcessor.java:1008)
        at org.apache.ignite.internal.processors.rest.GridRestProcessor.start(GridRestProcessor.java:558)
        at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1823)
        at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1105)
        at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1720)
        at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1642)
        at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1084)
        at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:978)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:884)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:803)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:673)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:642)
        at org.apache.ignite.Ignition.start(Ignition.java:325)
        at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:365)
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.core.JsonProcessingException
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
        at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
        ... 16 more

Environment Details:

  • Operating System: Tested on Kubernetes and Docker (AMD architecture)
  • Docker Image: apacheignite/ignite:2.17.0
  • Java Version: OpenJDK Runtime Environment 11.0.26+4 Eclipse Adoptium OpenJDK 64-Bit Server VM 11.0.26+4

Reproduction Steps:

Simpler reproduction steps with docker-compose is available in the following ticket: https://github.com/apache/ignite/issues/11877#issue-2853851199

  1. Create a Dockerfile with the following content:

    FROM apacheignite/ignite:2.17.0

ENV OPTION_LIBS=ignite-rest-http

  2. Build the Docker image:

    docker build -t testignite .

  3. Run the Docker container:

    docker run -it testignite

  4. Observe the error message and container crash:

    java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/JsonProcessingException

Expected Behavior:

Apache Ignite should start without any errors and the stack trace should not appear.

Additional Information:

  • The issue does not occur with apacheignite/ignite:2.16.0 using the same configuration and Dockerfile.

Possible Cause:

At the end of the logs, it also complains about USER_LIBS not being defined, maybe that's a clue ? The issue might be due to a missing dependency for the Jackson library in the apacheignite/ignite:2.17.0 Docker image.

Suggested Fix:

Investigate the Docker image apacheignite/ignite:2.17.0 to ensure that all necessary dependencies, including the Jackson library, are included. Alternatively, update the Dockerfile to include the missing dependency.

gammacomputer avatar Feb 18 '25 14:02 gammacomputer

Here is the complete logs I got in case it's useful:

WARNING: Unknown module: jdk.internal.jvmstat specified to --add-exports
WARNING: package jdk.internal.access not in java.base
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.ignite.internal.util.GridUnsafe$2 (file:/opt/ignite/apache-ignite/libs/ignite-core-2.17.0.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of org.apache.ignite.internal.util.GridUnsafe$2
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[14:20:38]    __________  ________________
[14:20:38]   /  _/ ___/ |/ /  _/_  __/ __/
[14:20:38]  _/ // (7 7    // /  / / / _/
[14:20:38] /___/\___/_/|_/___/ /_/ /x___/
[14:20:38]
[14:20:38] ver. 2.17.0#20250209-sha1:d53d4540
[14:20:38] 2025 Copyright(C) Apache Software Foundation
[14:20:38]
[14:20:38] Ignite documentation: https://ignite.apache.org
[14:20:38]
[14:20:38] Quiet mode.
[14:20:38]   ^-- Logging to file '/opt/ignite/apache-ignite/work/log/ignite-e2eea7dc.0.log'
[14:20:38]   ^-- Logging by 'JavaLogger [quiet=true, config=/opt/ignite/apache-ignite/config/java.util.logging.properties]'
[14:20:38]   ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or "-v" to ignite.{sh|bat}
[14:20:38]
[14:20:38] OS: Linux 5.15.146.1-microsoft-standard-WSL2 amd64
[14:20:38] VM information: OpenJDK Runtime Environment 11.0.26+4 Eclipse Adoptium OpenJDK 64-Bit Server VM 11.0.26+4
[14:20:38] Configured plugins:
[14:20:38]   ^-- None
[14:20:38]
[14:20:38] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]]]
[14:20:38] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides.
[14:20:38,755][SEVERE][main][IgniteKernal] Exception during start processors, node will be stopped and close connections
java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/JsonProcessingException
        at org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestProtocol.start(GridJettyRestProtocol.java:104)
        at org.apache.ignite.internal.processors.rest.GridRestProcessor.startProtocol(GridRestProcessor.java:1037)
        at org.apache.ignite.internal.processors.rest.GridRestProcessor.startHttpProtocol(GridRestProcessor.java:1008)
        at org.apache.ignite.internal.processors.rest.GridRestProcessor.start(GridRestProcessor.java:558)
        at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1823)
        at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1105)
        at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1720)
        at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1642)
        at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1084)
        at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:978)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:884)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:803)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:673)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:642)
        at org.apache.ignite.Ignition.start(Ignition.java:325)
        at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:365)
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.core.JsonProcessingException
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
        at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
        ... 16 more
[14:20:38,757][SEVERE][main][IgniteKernal] Got exception while starting (will rollback startup routine).
java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/JsonProcessingException
        at org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestProtocol.start(GridJettyRestProtocol.java:104)
        at org.apache.ignite.internal.processors.rest.GridRestProcessor.startProtocol(GridRestProcessor.java:1037)
        at org.apache.ignite.internal.processors.rest.GridRestProcessor.startHttpProtocol(GridRestProcessor.java:1008)
        at org.apache.ignite.internal.processors.rest.GridRestProcessor.start(GridRestProcessor.java:558)
        at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1823)
        at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1105)
        at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1720)
        at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1642)
        at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1084)
        at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:978)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:884)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:803)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:673)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:642)
        at org.apache.ignite.Ignition.start(Ignition.java:325)
        at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:365)
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.core.JsonProcessingException
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
        at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
        ... 16 more
[14:20:38] Ignite node stopped OK [uptime=00:00:03.696]
java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/JsonProcessingException
        at org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestProtocol.start(GridJettyRestProtocol.java:104)
        at org.apache.ignite.internal.processors.rest.GridRestProcessor.startProtocol(GridRestProcessor.java:1037)
        at org.apache.ignite.internal.processors.rest.GridRestProcessor.startHttpProtocol(GridRestProcessor.java:1008)
        at org.apache.ignite.internal.processors.rest.GridRestProcessor.start(GridRestProcessor.java:558)
        at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1823)
        at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1105)
        at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1720)
        at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1642)
        at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1084)
        at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:978)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:884)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:803)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:673)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:642)
        at org.apache.ignite.Ignition.start(Ignition.java:325)
        at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:365)
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.core.JsonProcessingException
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
        at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
        ... 16 more
Failed to start grid: com/fasterxml/jackson/core/JsonProcessingException
Note! You may use 'USER_LIBS' environment variable to specify your classpath.

I will investigate if the issue might be caused by USER_LIBS not being defined. Thanks a lot in advance for your help.

gammacomputer avatar Feb 18 '25 14:02 gammacomputer

As mentionned by Wolfteam, it seems that the issue is more likely caused by OPTION_LIBS=ignite-rest-http than CONFIG_URI, as the error is triggered with the following dockerfile:

FROM apacheignite/ignite:2.17.0

ENV OPTION_LIBS=ignite-rest-http

Initial comment updated with this information

gammacomputer avatar Feb 18 '25 15:02 gammacomputer

I tried the following dockerfile but there is the same issue:

FROM apacheignite/ignite:2.17.0

RUN cp -r ${IGNITE_HOME}/libs/optional/ignite-rest-http ${IGNITE_HOME}/libs/
RUN wget -P ${IGNITE_HOME}/libs/ https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.18.1/jackson-core-2.18.1.jar
ENV USER_LIBS=${IGNITE_HOME}/libs/

gammacomputer avatar Feb 18 '25 15:02 gammacomputer

Hello, @gammacomputer! As a workaround, you can add the ignite-json module to the Ignite classpath, similar to the ignite-rest-http module.

NSAmelchev avatar Feb 18 '25 16:02 NSAmelchev

The following workaround is working:

FROM apacheignite/ignite:2.17.0

RUN cp -r ${IGNITE_HOME}/libs/optional/ignite-rest-http ${IGNITE_HOME}/libs/
RUN cp -r ${IGNITE_HOME}/libs/optional/ignite-json ${IGNITE_HOME}/libs/

Thanks a lot @NSAmelchev !

gammacomputer avatar Feb 18 '25 16:02 gammacomputer

I'm not sure if I should close the ticket or not ? Maybe this behavior is expected to avoid having to load too much things ? Maybe what could be done would just update the documentation to tell that the module ignite-rest-http requires ignite-json ? https://ignite.apache.org/docs/latest/restapi

gammacomputer avatar Feb 18 '25 16:02 gammacomputer

No, please don't close; this a real issue. REST is a standard requirement in modern stacks. We should include it as a standard.

mingfang avatar Apr 05 '25 19:04 mingfang

It seems that it is enough to add information to the documentation that you additionally need to add the ignite-json module. I found some pages where it might be helpful:

  • https://ignite.apache.org/docs/latest/restapi
  • https://ignite.apache.org/docs/latest/quick-start/restapi
  • https://ignite.apache.org/docs/latest/logging (if you want JSON format in logs)

Scorcher avatar May 09 '25 16:05 Scorcher

Created a ticket for adding documentation and useful log if required module is missing. Thanks for reporting the issue

https://issues.apache.org/jira/browse/IGNITE-26385

timoninmaxim avatar Sep 03 '25 20:09 timoninmaxim