Grizzly http 2.3.20 instrumentation tests are flaky

[PerfectSlayer]

Test suite

test blocking of request for request body variant #variant from GrizzlyTest. Marked flaky by CircleCI

Error

Condition not satisfied:

spans.find { it.error && it.tags['error.type'] == BlockingException.name } != null
|     |                                                                    |
|     null                                                                 false
[DDSpan [ t_id=279, s_id=278, p_id=0 ] trace=worker.org.gradle.process.internal.worker.GradleWorkerMain/grizzly.request/POST /body-json *errored* *measured* tags={_dd.agent_psr=1.0, _dd.profiling.ctx=test, _dd.profiling.enabled=0, _dd.trace_span_attribute_schema=0, _sample_rate=1, appsec.blocked=true, component=grizzly-filterchain-server, error.message=Blocked request (for ReaderInterceptorExecutor/proceed), error.stack=datadog.appsec.api.blocking.BlockingException: Blocked request (for ReaderInterceptorExecutor/proceed)
	at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:159)
	at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundReadFrom(MappableExceptionWrapperInterceptor.java:74)
	at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:156)
	at org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1085)
	at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:853)
	at org.glassfish.jersey.server.ContainerRequest.readEntity(ContainerRequest.java:270)
	at org.glassfish.jersey.server.internal.inject.EntityParamValueFactoryProvider$EntityValueFactory.provide(EntityParamValueFactoryProvider.java:96)
	at org.glassfish.jersey.server.spi.internal.ParameterValueHelper.getParameterValues(ParameterValueHelper.java:81)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$AbstractMethodParamInvoker.getParamValues(JavaResourceMethodDispatcherProvider.java:127)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
	at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:308)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:291)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1140)
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:375)
	at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:206)
	at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:180)
	at org.glassfish.grizzly.http.server.HttpHandlerChain.doHandle(HttpHandlerChain.java:197)
	at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235)
	at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:283)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:132)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:111)
	at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
	at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:536)
	at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
	at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:103)
	at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89)
	at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:414)
	at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:383)
	at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:347)
	at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:278)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571)
	at java.base/java.lang.Thread.run(Thread.java:829)
, error.type=datadog.appsec.api.blocking.BlockingException, http.client_ip=127.0.0.1, http.hostname=localhost, http.method=POST, http.status_code=413, http.url=http://localhost:36829/body-json, http.useragent=okhttp/3.12.12, language=jvm, pathway.hash=16882414545893174995, peer.ipv4=127.0.0.1, peer.port=44604, process_id=5347, request.body={"a": "x"}, request.body.converted=[a:[x]], runtime-id=9f5fef5f-1ca2-4aaf-b2eb-64e9dc3b4f0a, span.kind=server, thread.id=270, thread.name=grizzly-nio-kernel(4) SelectorRunner}, duration_ns=7159729, forceKeep=false, links=[]]

	at datadog.trace.agent.test.base.HttpServerTest.test blocking of request for request body variant #variant(HttpServerTest.groovy:1676)

Cause

Span indicated request was blocked can’t be found. Surprisingly:

• Span seems to be in error according SpanContext.toString() (*errored* label),
• The error.type tag is present with the expected datadog.appsec.api.blocking.BlockingException.

This test was enabled in 55ba95a1.

Environment

Any version of Java

Logs

grizzly-http-2.3.20.zip

[PerfectSlayer]

I am not sure how to mark this test as @Flaky as the feature comes from a parent Specification (HttpServerTest) 🤷