vertx-tracing icon indicating copy to clipboard operation
vertx-tracing copied to clipboard

Can't get OpenTelemetry Context in different thread

Open semistone opened this issue 1 year ago • 6 comments

Questions

I am using vertx webclient in different thread with attach Context

VertxContextStorageProvider.VertxContextStorage.INSTANCE.attach(context);

when PREPARE_REQUEST and CREATE_REQUEST phase, it's still in original thread and still could get Context by

Context.current() 

at this phase, Vertx Context still doesn't exist, so it get context current from OpenTelemetry Context

but in SEND_REQUEST phase, it switch to different thread after create connection. and in this phase will create Vertx Context by vertx.getOrCreateContext() then call

OpenTelemetryTracer.sendRequest

it will try to get OpenTelemetry Context by

VertxContextStorage.INSTANCE.current();

and at this moment, Vertx Context is empty , so it can't get OpenTelemetry Context

currently, I wrote a Handler to propagator OpenTelemetry Context to SEND_REQUEST phase, but not sure is there any other better solution?

Version

4.5.7

Context

Do you have a reproducer?

I changed unit test in vertx-opentelemetry/src/test/java/io/vertx/tracing/opentelemetry/OpenTelemetryIntegrationTest.java

  • https://github.com/semistone/vertx-tracing/commit/96d439e0293bed238a036f807dcf050eae2e2247

Steps to reproduce

  1. run client in new thread
  2. attach Context to current thread
  3. call vertx webclient request

Extra

semistone avatar Jul 02 '24 02:07 semistone

currently, I wrote a Handler to propagator OpenTelemetry Context to SEND_REQUEST phase, but not sure is there any other better solution?

Can you paste here a snippet with your solution?

tsegismont avatar Jul 02 '24 13:07 tsegismont

currently, I wrote a Handler to propagator OpenTelemetry Context to SEND_REQUEST phase, but not sure is there any other better solution?

Can you paste here a snippet with your solution?

just put Context in httpContext and get it in SEND_REQUEST phase

c.makeCurrent() will save context into VertxContextStorage


	@Override
	public void handle(HttpContext<T> httpContext) {
		try {
			if (httpContext.phase() == ClientPhase.PREPARE_REQUEST) {
				httpContext.set(OPEN_TELEMETRY_CONTEXT, Context.current());
			}
			// send phase need to use PREPARE_REQUEST phase's context
			if (httpContext.phase() == ClientPhase.SEND_REQUEST
					&& httpContext.get(OPEN_TELEMETRY_CONTEXT) instanceof Context c && c != Context.current()) {
				httpContext.set(OPEN_TELEMETRY_SCOPE, c.makeCurrent());
			}
			if ((httpContext.phase() == ClientPhase.RECEIVE_RESPONSE || httpContext.phase() == ClientPhase.FAILURE)
					&& httpContext.get(OPEN_TELEMETRY_SCOPE) instanceof Scope s) {
				s.close();
			}
			httpContext.next();
		} catch (Exception e) {
			log.warn("switch OpenTelemetry Context fail", e);
		}
	}

semistone avatar Jul 03 '24 05:07 semistone

@semistone I can't think of anything much better than to be honest.

I wonder if you have tried OTel instrumentations like this one: https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/vertx/vertx-http-client/vertx-http-client-4.0/javaagent/src/main/java/io/opentelemetry/javaagent/instrumentation/vertx/v4_0/client

tsegismont avatar Jul 03 '24 08:07 tsegismont

@semistone I can't think of anything much better than to be honest.

I wonder if you have tried OTel instrumentations like this one: https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/vertx/vertx-http-client/vertx-http-client-4.0/javaagent/src/main/java/io/opentelemetry/javaagent/instrumentation/vertx/v4_0/client

I will try it later, but I guess those instrument will still rely on those Context to pass successfully.

semistone avatar Jul 03 '24 08:07 semistone

@semistone I can't think of anything much better than to be honest. I wonder if you have tried OTel instrumentations like this one: https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/vertx/vertx-http-client/vertx-http-client-4.0/javaagent/src/main/java/io/opentelemetry/javaagent/instrumentation/vertx/v4_0/client

I will try it later, but I guess those instrument will still rely on those Context to pass successfully.

I tested and it work

https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/vertx/vertx-http-client/vertx-http-client-4.0/javaagent/src/main/java/io/opentelemetry/javaagent/instrumentation/vertx/v4_0/client/HandlerWrapper.java

semistone avatar Jul 05 '24 01:07 semistone

Thanks for sharing this information with us @semistone !

I'm closing this since we have a good workaround (thanks to you) documented here and we know using the otel instrumentation solves the problem.

tsegismont avatar Jul 11 '24 09:07 tsegismont