ERROR co.elastic.apm.agent.bci.IndyBootstrap - Advice threw an exception, this should never happen!

[dbalijapalli0264]

Describe the bug

Hello Elastic team, Thank you for such an amazing product. I am opening this bug request to see if we can get some help from you on the error we are getting, while implementing elastic-apm-agent-1.45.0.jar or even the latest version elastic-apm-agent-1.47.0.jar. That's available on : https://mvnrepository.com/artifact/co.elastic.apm/elastic-apm-agent.

Our production log files are loaded with these 2 errors when we use the elastic-apm-agent-xxx.jar: 2024-02-15 20:12:34,254 [qtp292545106-1545] ERROR co.elastic.apm.agent.bci.IndyBootstrap - Advice threw an exception, this should never happen! and java.lang.ArrayIndexOutOfBoundsException: null

Steps to reproduce

Download teh api package from https://mvnrepository.com/artifact/co.elastic.apm/apm-agent-api/1.45.0 (or even 1.47.0)

2. Update corresponding code block to extract following information and expose to GTM data layer

Add api jar dependency

import co.elastic.apm.api.ElasticApm; import co.elastic.apm.api.Transaction;

Transaction transaction = ElasticApm.currentTransaction();

From transaction object we extract following attributes and store with “apmTransaction” key

traceId = transaction.traceId

spanId=transaction.ensureParentId()

sampled=transaction.sampled

Refer: https://www.elastic.co/guide/en/apm/agent/java/current/public-api.html

Expected behavior

Should not spam the logs with these errors: ERROR co.elastic.apm.agent.bci.IndyBootstrap - Advice threw an exception, this should never happen! AND java.lang.ArrayIndexOutOfBoundsException: null

Debug logs

Click to expand

replace this line with your debug logs

[JonasKunz]

Hi @dbalijapalli0264 , those two exceptions should also have stacktraces in addition to the exception messages. Could you please provide the stacktraces?

[dbalijapalli0264]

Apologies for not getting back earlier. I don't see any stack trace for the error that says: "ERROR co.elastic.apm.agent.bci.IndyBootstrap - Advice threw an exception, this should never happen!"

But do see the below for the java.lang.ArrayIndexOutOfBoundsException: null

java.lang.ArrayIndexOutOfBoundsException: Index 34 out of bounds for length 34 at java.lang.AbstractStringBuilder.charAt(AbstractStringBuilder.java:353) ~[?:?] at java.lang.StringBuilder.charAt(StringBuilder.java:91) ~[?:?] at co.elastic.apm.agent.metrics.Labels$AbstractBase.hash(Labels.java:201) ~[elastic-apm-agent-1.43.0.jar:1.43.0] at co.elastic.apm.agent.metrics.Labels$AbstractBase.hashCode(Labels.java:135) ~[elastic-apm-agent-1.43.0.jar:1.43.0] at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936) ~[?:?] at co.elastic.apm.agent.metrics.MetricRegistry.getOrCreateMetricSet(MetricRegistry.java:233) ~[elastic-apm-agent-1.43.0.jar:1.43.0] at co.elastic.apm.agent.metrics.MetricRegistry.updateTimer(MetricRegistry.java:218) ~[elastic-apm-agent-1.43.0.jar:1.43.0] at co.elastic.apm.agent.impl.transaction.Transaction.trackMetrics(Transaction.java:525) ~[elastic-apm-agent-1.43.0.jar:1.43.0] at co.elastic.apm.agent.impl.transaction.Transaction.afterEnd(Transaction.java:284) ~[elastic-apm-agent-1.43.0.jar:1.43.0] at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:580) ~[elastic-apm-agent-1.43.0.jar:1.43.0] at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:557) ~[elastic-apm-agent-1.43.0.jar:1.43.0] at co.elastic.apm.agent.servlet.ServletTransactionHelper.onAfter(ServletTransactionHelper.java:248) ~[elastic-apm-agent-1.43.0.jar:1.43.0] at co.elastic.apm.agent.servlet.ServletApiAdvice.onExitServlet(ServletApiAdvice.java:253) ~[elastic-apm-agent-1.43.0.jar:1.43.0] at co.elastic.apm.agent.servlet.JavaxServletApiAdvice.onExitServletService(JavaxServletApiAdvice.java:45) ~[elastic-apm-agent-1.43.0.jar:1.43.0] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1602) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at com.about.mantle.spring.jetty.JettyBaseConfig$GlobeGzipHandler.handle(JettyBaseConfig.java:436) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:722) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at com.about.mantle.spring.jetty.JettyBaseConfig$GlobeGzipHandler.handle(JettyBaseConfig.java:436) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at com.about.mantle.spring.jetty.MantleInstrumentedHandler.handle(MantleInstrumentedHandler.java:92) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400) ~[people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645) [people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392) [people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) [people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [people-launcher.jar:1.175.0-FEATURE-PEOB-468-CHECKOUTMANTLE3-14-431-TESTINGAPM-SNAPSHOT] at java.lang.Thread.run(Thread.java:833) [?:?]

[dbalijapalli0264]

stack trace right before the error: java.lang.ArrayIndexOutOfBoundsException

error.exception.stacktrace [ { "exclude_from_grouping": false, "filename": "DebugHttpServletRequest.java", "classname": "com.about.globe.core.http.DebugHttpServletRequest", "line": { "number": 166 }, "function": "getParameterMap", "module": "com.about.globe.core.http" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "HttpRequestContextContributor.java", "classname": "com.about.mantle.http.HttpRequestContextContributor", "line": { "number": 188 }, "function": "contribute", "module": "com.about.mantle.http" }, { "exclude_from_grouping": false, "filename": "RequestContextFactoryImpl.java", "classname": "com.about.globe.core.http.impl.RequestContextFactoryImpl", "line": { "number": 24 }, "function": "create", "module": "com.about.globe.core.http.impl" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "MantleDebugRequestHandlerMethods.java", "classname": "com.about.mantle.handlers.methods.MantleDebugRequestHandlerMethods", "line": { "number": 176 }, "function": "debugSupportMethod", "module": "com.about.mantle.handlers.methods" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "MantleDebugRequestHandlerMethods.java", "classname": "com.about.mantle.handlers.methods.MantleDebugRequestHandlerMethods", "line": { "number": 147 }, "function": "debugModels", "module": "com.about.mantle.handlers.methods" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "DebugModelsController.java", "classname": "com.about.mantle.endpoint.controllers.DebugModelsController", "line": { "number": 23 }, "function": "doGet", "module": "com.about.mantle.endpoint.controllers" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "AbstractMantleEndpointController.java", "classname": "com.about.mantle.endpoint.controllers.AbstractMantleEndpointController", "line": { "number": 73 }, "function": "doService", "module": "com.about.mantle.endpoint.controllers" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "AbstractMantleEndpointController.java", "classname": "com.about.mantle.endpoint.controllers.AbstractMantleEndpointController", "line": { "number": 114 }, "function": "handleRequest", "module": "com.about.mantle.endpoint.controllers" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "SimpleControllerHandlerAdapter.java", "classname": "org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter", "line": { "number": 52 }, "function": "handle", "module": "org.springframework.web.servlet.mvc" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "DispatcherServlet.java", "classname": "org.springframework.web.servlet.DispatcherServlet", "line": { "number": 1060 }, "function": "doDispatch", "module": "org.springframework.web.servlet" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "DispatcherServlet.java", "classname": "org.springframework.web.servlet.DispatcherServlet", "line": { "number": 962 }, "function": "doService", "module": "org.springframework.web.servlet" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "FrameworkServlet.java", "classname": "org.springframework.web.servlet.FrameworkServlet", "line": { "number": 1006 }, "function": "processRequest", "module": "org.springframework.web.servlet" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "FrameworkServlet.java", "classname": "org.springframework.web.servlet.FrameworkServlet", "line": { "number": 898 }, "function": "doGet", "module": "org.springframework.web.servlet" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "HttpServlet.java", "classname": "javax.servlet.http.HttpServlet", "line": { "number": 687 }, "function": "service", "module": "javax.servlet.http" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "FrameworkServlet.java", "classname": "org.springframework.web.servlet.FrameworkServlet", "line": { "number": 883 }, "function": "service", "module": "org.springframework.web.servlet" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "HttpServlet.java", "classname": "javax.servlet.http.HttpServlet", "line": { "number": 790 }, "function": "service", "module": "javax.servlet.http" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "ServletHolder.java", "classname": "org.eclipse.jetty.servlet.ServletHolder", "line": { "number": 799 }, "function": "handle", "module": "org.eclipse.jetty.servlet" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "ServletHandler.java", "classname": "org.eclipse.jetty.servlet.ServletHandler$ChainEnd", "line": { "number": 1631 }, "function": "doFilter", "module": "org.eclipse.jetty.servlet" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "WebSocketUpgradeFilter.java", "classname": "org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter", "line": { "number": 228 }, "function": "doFilter", "module": "org.eclipse.jetty.websocket.server" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "FilterHolder.java", "classname": "org.eclipse.jetty.servlet.FilterHolder", "line": { "number": 193 }, "function": "doFilter", "module": "org.eclipse.jetty.servlet" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "ServletHandler.java", "classname": "org.eclipse.jetty.servlet.ServletHandler$Chain", "line": { "number": 1601 }, "function": "doFilter", "module": "org.eclipse.jetty.servlet" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "ContentCachingRequestFilter.java", "classname": "com.about.mantle.web.filter.ContentCachingRequestFilter", "line": { "number": 23 }, "function": "doHttpFilter", "module": "com.about.mantle.web.filter" }, { "exclude_from_grouping": false, "filename": "AbstractCoreFilter.java", "classname": "com.about.globe.core.web.filter.AbstractCoreFilter", "line": { "number": 37 }, "function": "doFilter", "module": "com.about.globe.core.web.filter" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "FilterHolder.java", "classname": "org.eclipse.jetty.servlet.FilterHolder", "line": { "number": 193 }, "function": "doFilter", "module": "org.eclipse.jetty.servlet" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "ServletHandler.java", "classname": "org.eclipse.jetty.servlet.ServletHandler$Chain", "line": { "number": 1601 }, "function": "doFilter", "module": "org.eclipse.jetty.servlet" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "XContentTypeOptionsFilter.java", "classname": "com.about.mantle.web.filter.XContentTypeOptionsFilter", "line": { "number": 24 }, "function": "doHttpFilter", "module": "com.about.mantle.web.filter" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "AbstractHttpFilter.java", "classname": "com.about.mantle.web.filter.AbstractHttpFilter", "line": { "number": 31 }, "function": "doFilter", "module": "com.about.mantle.web.filter" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "FilterHolder.java", "classname": "org.eclipse.jetty.servlet.FilterHolder", "line": { "number": 193 }, "function": "doFilter", "module": "org.eclipse.jetty.servlet" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "ServletHandler.java", "classname": "org.eclipse.jetty.servlet.ServletHandler$Chain", "line": { "number": 1601 }, "function": "doFilter", "module": "org.eclipse.jetty.servlet" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "MalformedURLFilter.java", "classname": "com.about.mantle.web.filter.MalformedURLFilter", "line": { "number": 29 }, "function": "doHttpFilter", "module": "com.about.mantle.web.filter" }, { "exclude_from_grouping": false, "filename": "AbstractCoreFilter.java", "classname": "com.about.globe.core.web.filter.AbstractCoreFilter", "line": { "number": 37 }, "function": "doFilter", "module": "com.about.globe.core.web.filter" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "FilterHolder.java", "classname": "org.eclipse.jetty.servlet.FilterHolder", "line": { "number": 193 }, "function": "doFilter", "module": "org.eclipse.jetty.servlet" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "ServletHandler.java", "classname": "org.eclipse.jetty.servlet.ServletHandler$Chain", "line": { "number": 1601 }, "function": "doFilter", "module": "org.eclipse.jetty.servlet" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "ExternalServiceProxyFilter.java", "classname": "com.about.mantle.web.filter.ExternalServiceProxyFilter", "line": { "number": 36 }, "function": "doHttpFilter", "module": "com.about.mantle.web.filter" }, { "exclude_from_grouping": false, "filename": "AbstractCoreFilter.java", "classname": "com.about.globe.core.web.filter.AbstractCoreFilter", "line": { "number": 37 }, "function": "doFilter", "module": "com.about.globe.core.web.filter" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "FilterHolder.java", "classname": "org.eclipse.jetty.servlet.FilterHolder", "line": { "number": 193 }, "function": "doFilter", "module": "org.eclipse.jetty.servlet" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "ServletHandler.java", "classname": "org.eclipse.jetty.servlet.ServletHandler$Chain", "line": { "number": 1601 }, "function": "doFilter", "module": "org.eclipse.jetty.servlet" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "ResponseHeadersFilter.java", "classname": "com.about.mantle.web.filter.ResponseHeadersFilter", "line": { "number": 33 }, "function": "doFilter", "module": "com.about.mantle.web.filter" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "FilterHolder.java", "classname": "org.eclipse.jetty.servlet.FilterHolder", "line": { "number": 193 }, "function": "doFilter", "module": "org.eclipse.jetty.servlet" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "ServletHandler.java", "classname": "org.eclipse.jetty.servlet.ServletHandler$Chain", "line": { "number": 1601 }, "function": "doFilter", "module": "org.eclipse.jetty.servlet" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "SafelistHttpMethodFilter.java", "classname": "com.about.mantle.web.filter.SafelistHttpMethodFilter", "line": { "number": 23 }, "function": "doHttpFilter", "module": "com.about.mantle.web.filter" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "AbstractHttpFilter.java", "classname": "com.about.mantle.web.filter.AbstractHttpFilter", "line": { "number": 31 }, "function": "doFilter", "module": "com.about.mantle.web.filter" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "FilterHolder.java", "classname": "org.eclipse.jetty.servlet.FilterHolder", "line": { "number": 193 }, "function": "doFilter", "module": "org.eclipse.jetty.servlet" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "ServletHandler.java", "classname": "org.eclipse.jetty.servlet.ServletHandler$Chain", "line": { "number": 1601 }, "function": "doFilter", "module": "org.eclipse.jetty.servlet" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "EnsureHttpsFilter.java", "classname": "com.about.mantle.web.filter.EnsureHttpsFilter", "line": { "number": 33 }, "function": "doHttpFilter", "module": "com.about.mantle.web.filter" }, { "exclude_from_grouping": false, "filename": "AbstractCoreFilter.java", "classname": "com.about.globe.core.web.filter.AbstractCoreFilter", "line": { "number": 37 }, "function": "doFilter", "module": "com.about.globe.core.web.filter" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "FilterHolder.java", "classname": "org.eclipse.jetty.servlet.FilterHolder", "line": { "number": 193 }, "function": "doFilter", "module": "org.eclipse.jetty.servlet" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "ServletHandler.java", "classname": "org.eclipse.jetty.servlet.ServletHandler$Chain", "line": { "number": 1601 }, "function": "doFilter", "module": "org.eclipse.jetty.servlet" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "TraceFilter.java", "classname": "com.about.mantle.web.filter.TraceFilter", "line": { "number": 43 }, "function": "doFilter", "module": "com.about.mantle.web.filter" }, { "exclude_from_grouping": false, "library_frame": true, "filename": "FilterHolder.java", "classname": "org.eclipse.jetty.servlet.FilterHolder", "line": { "number": 193 }, "function": "doFilter", "module": "org.eclipse.jetty.servlet" } ]

[JonasKunz]

Are you using Transaction.setName anywhere in you code? The shown error indicates a race condition:

• The transaction name is internally stored in a StringBuilder
• When the transaction ends, a hashcode is computed for the transaction name (stored in the StringBuilder)
• If now the transaction name is updated concurrently after the transaction has ended, that can lead to the ArrayIndexOutOfBounds exception you are seeing

[dbalijapalli0264]

Thank you @JonasKunz, looking into it now. Will get back with the outcome.

[SylvainJuge]

Hi @dbalijapalli0264, there hasn't been any update since your last message, is this issue still happening in your environment ?

[jackshirazi]

Closing this, if it's still an issue, comment or reopen it as needed

[dbalijapalli0264]

Hello, Sorry for lack of response, we had to roll back the changes as this exception was overloading our production error logs. We are back testing it now. and i do see that the exception is still happening.

Here are the exceptions, these both exceptions happen together when they happen.

co.elastic.apm.agent.bci.IndyBootstrap

message:2024-05-09 18:20:20,202 [qtp2056288620-1138] ERROR co.elastic.apm.agent.bci.IndyBootstrap - Advice threw an exception, this should never happen!

java.lang.ArrayIndexOutOfBoundsException

java.lang.ArrayIndexOutOfBoundsException: Index 34 out of bounds for length 34 at java.lang.AbstractStringBuilder.charAt(AbstractStringBuilder.java:353) ~[?:?] at java.lang.StringBuilder.charAt(StringBuilder.java:91) ~[?:?] at co.elastic.apm.agent.metrics.Labels$AbstractBase.hash(Labels.java:201) ~[elastic-apm-agent-1.47.0.jar:1.47.0] at co.elastic.apm.agent.metrics.Labels$AbstractBase.hashCode(Labels.java:135) ~[elastic-apm-agent-1.47.0.jar:1.47.0] at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936) ~[?:?] at co.elastic.apm.agent.metrics.MetricRegistry.getOrCreateMetricSet(MetricRegistry.java:233) ~[elastic-apm-agent-1.47.0.jar:1.47.0] at co.elastic.apm.agent.metrics.MetricRegistry.updateTimer(MetricRegistry.java:218) ~[elastic-apm-agent-1.47.0.jar:1.47.0] at co.elastic.apm.agent.impl.transaction.Transaction.trackMetrics(Transaction.java:525) ~[elastic-apm-agent-1.47.0.jar:1.47.0] at co.elastic.apm.agent.impl.transaction.Transaction.afterEnd(Transaction.java:284) ~[elastic-apm-agent-1.47.0.jar:1.47.0] at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:578) ~[elastic-apm-agent-1.47.0.jar:1.47.0] at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:548) ~[elastic-apm-agent-1.47.0.jar:1.47.0] at co.elastic.apm.agent.servlet.ServletTransactionHelper.onAfter(ServletTransactionHelper.java:248) ~[elastic-apm-agent-1.47.0.jar:1.47.0] at co.elastic.apm.agent.servlet.ServletApiAdvice.onExitServlet(ServletApiAdvice.java:253) ~[elastic-apm-agent-1.47.0.jar:1.47.0] at co.elastic.apm.agent.servlet.JavaxServletApiAdvice.onExitServletService(JavaxServletApiAdvice.java:45) ~[elastic-apm-agent-1.47.0.jar:1.47.0]

[dbalijapalli0264]

Could we reopen this issue please? SylvainJuge or jackshirazi

[SylvainJuge]

Hi @dbalijapalli0264 , as Jonas asked previously here do you know if the transaction name is modified using Transaction.setName

[SylvainJuge]

@dbalijapalli0264 to be a bit more clear here, we need more information about your context in order to make any progress here, otherwise there is no point in keeping this open if we don't have any actionable insights on the things we might need to fix here.

[dbalijapalli0264]

Hello, and apologies, that it took me a while before i got back the information, here is the information you requested.

@SylvainJuge or @jackshirazi your question: Are you using Transaction.setName --- yes we are setting the transaction name. Bt we are setting the name before the transaction ends. The transaction name is not updated concurrently after the transaction has ended.

Below is the snippet of our code. The backend to frontend correlation works fine(attached an image, please see), but, our logs are over loaded with the ArrayIndexOutOfBounds and co.elastic.apm.agent.bci.IndyBootstrap errors, which is not acceptable.

(PS: Please note in the code below, we don't set the transaction name for the deferred calls. )

---

import co.elastic.apm.api.ElasticApm;
import co.elastic.apm.api.Transaction;
import com.about.mantle.model.gtm.ApmTransaction;

public ApmTransaction createApmTransaction(RequestContext rc) {
if (rc.isDeferred()) {
    return null;
}

ApmTransaction.Builder builder = new ApmTransaction.Builder();
try {
    Transaction apmTransaction = ElasticApm.currentTransaction();
    builder = builder.traceId(apmTransaction.getTraceId()).spanId(apmTransaction.ensureParentId()).sampled(apmTransaction.isSampled());
    apmTransaction.setName(rc.getTemplateName());
} catch (Exception e) {
    log.error("Error adding elastic apm transaction to gtm model", e);
}
return builder.build();
}

---

From the attached screenshot, as you see, we are just trying to set the template name to the transaction. without the above code to set transaction name, the template name is not being set.

If this is not the proper way to do it, could you please let me know how to do it so that we have a proper correlation from backend to frontend. (PS: Its been very frustrating that we are not able to get this work fine despite all our efforts to have a proper correlated transaction.)

[JonasKunz]

But we are setting the name before the transaction ends. The transaction name is not updated concurrently after the transaction has ended.

Unfortunately that is the only possible explanation for the error you are seeing I can come up with right now. Also we have never seen this error occur before. Could you remove all calls to Transaction.setName and check whether the error still occurs?

[dbalijapalli0264]

Thanks you for the response @JonasKunz, Checking with our team internally, will get back to you.

[dbalijapalli0264]

We are trying to remove the Transaction.setName after running a set of tests, please keep this ticket active for a bit longer.

[paulbandler]

Was there any closure on this issue? I've just come across the same exception. My application is a grails application. We have no APM API calls in our application, buy I believe the apm agent is 'grails aware' in that it identifies transactions with grails controller entry points, so perhaps it is internally setting the transaction name as mentioned above?

This has started to happen as I was increasing the load today.

[jackshirazi]

please open a discussion item at our forum