Jaeger Middleware: Missing Span Context in Proxied Request Header

[jamesstocktonj1]

Description

I tried to create a simple Echo reverse proxy (echo/v4/middleware) which connects to 2 simple Echo services (Hello World), both with the Jaeger middleware connecting to the jaeger/all-in-one docker image. I noticed that each of the services was creating their own trace in Jaeger rather than the proxy propagating through to the hello world instance. The correct headers were not being set for the context to be propagated.

Before:

james@james-desktop:~$ docker logs jaeger-server1 --follow

   ____    __
  / __/___/ /  ___
 / _// __/ _ \/ _ \
/___/\__/_//_/\___/ v4.11.4
High performance, minimalist Go web framework
https://echo.labstack.com
____________________________________O/_______
                                    O\
⇨ http server started on [::]:8080
Context: <nil>
Error: opentracing: SpanContext not found in Extract carrier
Creating New Span Context
Request Header: map[Accept:[*/*] Accept-Encoding:[gzip] User-Agent:[curl/7.81.0] X-Forwarded-For:[192.168.96.1] X-Forwarded-Proto:[http] X-Real-Ip:[192.168.96.1]]
{"time":"2024-02-17T19:34:10.364168015Z","id":"","remote_ip":"192.168.96.1","host":"localhost:8080","method":"GET","uri":"/","user_agent":"curl/7.81.0","status":200,"error":"","latency":87721,"latency_human":"87.721µs","bytes_in":0,"bytes_out":13}

With the default settings, the config.Tracer.Extract function at the start of the Jaeger middleware is looking for the trace context in the header value "uber-trace-id" (jaeger.TraceContextHeaderName). This fix adds the context value in the header by using the config.Tracer.Inject function.

After:

james@james-desktop:~$ docker logs jaeger-server1 --follow

   ____    __
  / __/___/ /  ___
 / _// __/ _ \/ _ \
/___/\__/_//_/\___/ v4.11.4
High performance, minimalist Go web framework
https://echo.labstack.com
____________________________________O/_______
                                    O\
⇨ http server started on [::]:8080
Context: 08a020d7daedd7d2:08a020d7daedd7d2:0000000000000000:1
Error: <nil>
Use Existing Span Context
Request Header: map[Accept:[*/*] Accept-Encoding:[gzip] Uber-Trace-Id:[08a020d7daedd7d2:129bd5d44715283c:08a020d7daedd7d2:1] User-Agent:[curl/7.81.0] X-Forwarded-For:[192.168.96.1] X-Forwarded-Proto:[http] X-Real-Ip:[192.168.96.1]]
{"time":"2024-02-17T19:34:43.39425835Z","id":"","remote_ip":"192.168.96.1","host":"localhost:8080","method":"GET","uri":"/","user_agent":"curl/7.81.0","status":200,"error":"","latency":114142,"latency_human":"114.142µs","bytes_in":0,"bytes_out":13}

See also: serializing-to-the-wire

TLDR

Fix Jaeger middleware when used with a proxy by adding the trace context to the request header.

[jamesstocktonj1]

Hi @carlosedp can you please review?

[aldas]

@jamesstocktonj1 could you provide example for for these 2 small apps and docker command to run jaeger. I have no experience with Jaeger and I would like to avoid merging things without even being able to see/test them out myself.

[carlosedp]

Hi James, thanks for tagging me but I've been a little away from Echo and Go recently...

[jamesstocktonj1]

Hi all, thanks for the reply. I have created a quick project to demonstrate the issue.

[aldas]

Thank you @jamesstocktonj1 Your example project was helpful. I did not have Jaeger experience.

For history sake for myself. These were steps to reproduce. I simplified that example project to

1. start Jaeger docker image in host network mode just to get away from port mappings

docker run --rm -d --network host --name my_jaeger jaegertracing/all-in-one:latest

2. Running proxy in my IDE and settings env variables in program

package main

import (
	"log"
	"net/url"
	"os"

	"github.com/labstack/echo-contrib/jaegertracing"
	"github.com/labstack/echo/v4"
	"github.com/labstack/echo/v4/middleware"
)

// docker run --rm -d --network host --name my_jaeger jaegertracing/all-in-one:latest
func main() {
	os.Setenv("JAEGER_AGENT_HOST", "localhost")
	os.Setenv("JAEGER_AGENT_PORT", "6831")
	os.Setenv("JAEGER_SERVICE_NAME", "echo-proxy-1")

	// basic echo server
	s := echo.New()
	s.Use(middleware.Logger())
	s.Use(middleware.Recover())

	// init jaeger middleware
	closer := jaegertracing.New(s, nil)
	defer closer.Close()

	// init proxy middleware
	urlString := []string{
		"http://localhost:8081",
	}
	urls, err := parseUrls(urlString)
	if err != nil {
		log.Fatal(err)
	}
	s.Use(middleware.Proxy(middleware.NewRoundRobinBalancer(urls)))

	// start server
	s.Logger.Fatal(s.Start(":8080"))
}

func parseUrls(urls []string) ([]*middleware.ProxyTarget, error) {
	var targets []*middleware.ProxyTarget
	for _, u := range urls {
		u, err := url.Parse(u)
		if err != nil {
			return nil, err
		}
		targets = append(targets, &middleware.ProxyTarget{URL: u})
	}
	return targets, nil
}

3. Running server in my IDE and settings env variable in program

package main

import (
	"net/http"
	"os"

	"github.com/labstack/echo-contrib/jaegertracing"
	"github.com/labstack/echo/v4"
	"github.com/labstack/echo/v4/middleware"
)

// docker run --rm -d --network host --name my_jaeger jaegertracing/all-in-one:latest
func main() {
	os.Setenv("JAEGER_AGENT_HOST", "localhost")
	os.Setenv("JAEGER_AGENT_PORT", "6831")
	os.Setenv("JAEGER_SERVICE_NAME", "echo-server")

	// basic echo server
	s := echo.New()
	s.Use(middleware.Logger())
	s.Use(middleware.Recover())

	// init jaeger middleware
	closer := jaegertracing.New(s, nil)
	defer closer.Close()

	s.GET("/hello", func(c echo.Context) error {
		return c.String(http.StatusOK, "Hello, World!")
	})

	// start server
	s.Logger.Fatal(s.Start(":8081"))
}

4. Executing request to proxy

curl localhost:8080/hello

5. Opening Jaeger UI http://localhost:16686/ and selecting echo-proxy-1 as service and clicking "Find traces"

6. Stoppings proxy and commenting out that PR line and trying CURL again.

[aldas]

I have tagged new release. https://github.com/labstack/echo-contrib/releases/tag/v0.17.0