Jaeger exporter sending oversized UDP thrift packets

[Baliedge]

trafficstars

Description

When Jaeger exporter is configured to export to a remote host, significant portions of traces are incomplete or completely missing. Observing with tcpdump -i any port 6831 reveals many errors about packet size larger than the interface's 1500 MTU and consequently dropped.

The root cause is the exporter's MaxPacketSize is defaulted to 65000 which is only compatible via loopback interface to localhost. All other network interfaces are likely the standard 1500 MTU.

Workaround is to add client side logic to instantiate the Jaeger exporter with the appropriate MaxPacketSize to match the MTU. e.g.:

var agentEndpointOpts []jaeger.AgentEndpointOption

agentHost := os.Getenv("OTEL_EXPORTER_JAEGER_AGENT_HOST")
if agentHost != "" && agentHost != "localhost" && agentHost != "127.0.0.1" {
	// Default MaxPacketSize=65000, which only works with Jaeger agent on
	// localhost (loopback interface).
	// For tracing over network, packets must fit in MTU 1500, which has a
	// payload size of 1472.
	agentEndpointOpts = append(agentEndpointOpts, jaeger.WithMaxPacketSize(1472))
}

exp, err := jaeger.New(
	jaeger.WithAgentEndpoint(agentEndpointOpts...),
)

My expectation was that the client should not be concerned with MTU size when instantiating the exporter. While this could be an incorrect assumption, I didn't catch anything in docs explaining this possibility.

Environment

• OS: Occurs in Linux and Mac OS
• Architecture: amd64
• Go Version: 1.17
• opentelemetry-go version: v1.4.1

Steps To Reproduce

1. In a separate terminal run command: sudo tcpdump -i any port 6831
2. Instantiate tracer like so: (error handling omitted)

exp, _ := jaeger.New()
opts := []sdktrace.TracerProviderOption{
	sdktrace.WithBatcher(exp),
}
tp := sdktrace.NewTracerProvider(opts...)
otel.SetTracerProvider(tp)
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))

3. Create a trace:

tp, _ := otel.GetTracerProvider().(*sdktrace.TracerProvider)
tracer := tp.Tracer(libraryName)
ctx, span := tracer.Start(context.Background(), "My span")
defer span.Finish()

// ...

// On shutdown
tp, _ := otel.GetTracerProvider().(*sdktrace.TracerProvider)
tp.Shutdown(ctx)

4. Set environment variable:

• OTEL_EXPORTER_JAEGER_AGENT_HOST=myserver.example.com

5. Compile and run code containing above snippets.
6. Observe tcpdump output containing "bad length" errors:

16:08:29.522899 IP 10.0.0.21.50116 > myserver.example.com.ambit-lm: UDP, bad length 2918 > 1472

7. Observe that Jaeger UI shows the relevant trace is either missing or incomplete.

Expected behavior

Expected successful packet transmit output from tcpdump that looks like:

15:30:56.941562 IP 10.0.0.21.55660 > myserver.example.com.ambit-lm: UDP, length 710

And the Jaeger UI shows the trace with all expected child spans and metadata.

[MrAlias]

It looks like this originated in OpenCensus:

https://github.com/census-ecosystem/opencensus-go-exporter-jaeger/blob/30c8b0fe8ad9d0eac5785893f3941b2e72c5aaaa/agent.go#L28-L29

I'm not sure how it has persisted for so long if there is such a fundamental problem here (:shrug:).

Let's change the default to a reasonable and safe value. Based on this the safest value would likely be 508. This seems quite small and likely sub-optimal for the general (Ethernet or wireless) network use. Should we look into dynamically determining this value if unset? For example:

const absMaxDatagramSize = 2147483646 // 2**31-2

func getMaxDatagramSize() int {
	var m int = 65535
	ifs, err := net.Interfaces()
	if err != nil {
		for i := range ifs {
			if ifs[i].MTU > m {
				m = ifs[i].MTU
			}
		}
	}
	if m > absMaxDatagramSize {
		m = absMaxDatagramSize
	}
	return m
}

[MadVikingGod]

I did a bit of digital spelunking and I think this is the default because the Agent (and the thrift agent service) are intended to be used only locally. An agent should be running on the host machine so you can use the loopback device. The collector service uses HTTP side stepping this problem.

My recommendation would be to use a collector endpoint, or if using the agent protocol over a network, don't batch because it would be only 2-3 spans per batch anyways to stay under a safe size.

[deejgregor]

FYI: there is an option on the Python Jaeger thrift exporter, split_oversized_batches to split batches that are too large:

https://github.com/open-telemetry/opentelemetry-python/blob/main/exporter/opentelemetry-exporter-jaeger-thrift/src/opentelemetry/exporter/jaeger/thrift/send.py#L73

[pellared]

Jaeger exporter is removed. Please use OTLP exporter instead.