agent

agent copied to clipboard

I'm getting a crash when trying to introspect some discovered log targets using grafana agent's promtail config. Using a Docker image built with FROM grafana/agent:v0.27.1.

It happens with an extremely minimal config:

server:
  log_level: debug

logs:
  configs:
    - name: agent
      clients:
        - url: https://<LOGIN>@logs-prod3.grafana.net/loki/api/v1/push
      positions:
        filename: /var/run/grafana/log-positions.yaml
      target_config:
        sync_period: 10s
      scrape_configs:
        - job_name: containers
          docker_sd_configs:
            - host: unix:///var/run/docker.sock
              refresh_interval: 5s

The API request:

$ curl -v localhost:12345/agent/api/v1/logs/targets
*   Trying 127.0.0.1:12345...
* Connected to localhost (127.0.0.1) port 12345 (#0)
> GET /agent/api/v1/logs/targets HTTP/1.1
> Host: localhost:12345
> User-Agent: curl/7.81.0
> Accept: */*
>
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server

Traceback:

2022-09-18 16:28:42.428305 I | http: panic serving 127.0.0.1:43106: runtime error: invalid memory address or nil pointer dereference
goroutine 420581 [running]:
net/http.(*conn).serve.func1()
 /usr/local/go/src/net/http/server.go:1825 +0xbf
panic({0x444c820, 0x81cf790})
 /usr/local/go/src/runtime/panic.go:844 +0x258
github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5.1()
 /go/pkg/mod/github.com/opentracing-contrib/[email protected]/nethttp/server.go:150 +0x139
panic({0x444c820, 0x81cf790})
 /usr/local/go/src/runtime/panic.go:838 +0x207
github.com/grafana/loki/clients/pkg/promtail/targets/docker.(*Target).Details(0xc000c0ce70)
 /go/pkg/mod/github.com/grafana/[email protected]/clients/pkg/promtail/targets/docker/target.go:238 +0xa5
github.com/grafana/agent/pkg/logs.listTargetsHandler.func1({0x5a27fc0, 0xc0036dc330}, 0xc0005d12b0?)
 /src/agent/pkg/logs/http.go:59 +0x2cf
net/http.HandlerFunc.ServeHTTP(0xc0036dc420?, {0x5a27fc0?, 0xc0036dc330?}, 0x5?)
 /usr/local/go/src/net/http/server.go:2084 +0x2f
github.com/grafana/agent/pkg/logs.(*Logs).ListTargetsHandler(0x4094040?, {0x5a27fc0, 0xc0036dc330}, 0xc0001ccaa0?)
 /src/agent/pkg/logs/http.go:43 +0x154
net/http.HandlerFunc.ServeHTTP(0xc003680b00?, {0x5a27fc0?, 0xc0036dc330?}, 0xc000477740?)
 /usr/local/go/src/net/http/server.go:2084 +0x2f
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000bb2480, {0x5a27fc0, 0xc0036dc330}, 0xc003680a00)
 /go/pkg/mod/github.com/gorilla/[email protected]/mux.go:210 +0x1cf
github.com/weaveworks/common/middleware.Instrument.Wrap.func1.2({0x5a27fc0?, 0xc0036dc330?})
 /go/pkg/mod/github.com/weaveworks/[email protected]/middleware/instrument.go:70 +0x39
github.com/felixge/httpsnoop.CaptureMetricsFn({0x5a17b80, 0xc0036158b0}, 0xc000c556b0)
 /go/pkg/mod/github.com/felixge/[email protected]/capture_metrics.go:76 +0x204
github.com/weaveworks/common/middleware.Instrument.Wrap.func1({0x5a17b80, 0xc0036158b0}, 0xc003680a00)
 /go/pkg/mod/github.com/weaveworks/[email protected]/middleware/instrument.go:69 +0x35b
net/http.HandlerFunc.ServeHTTP(0xc003680a00?, {0x5a17b80?, 0xc0036158b0?}, 0x0?)
 /usr/local/go/src/net/http/server.go:2084 +0x2f
github.com/weaveworks/common/middleware.Log.Wrap.func1({0x5a283e0?, 0xc003615860}, 0xc003680a00)
 /go/pkg/mod/github.com/weaveworks/[email protected]/middleware/logging.go:55 +0x287
net/http.HandlerFunc.ServeHTTP(0x414207?, {0x5a283e0?, 0xc003615860?}, 0x8511101?)
 /usr/local/go/src/net/http/server.go:2084 +0x2f
github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5({0x5a25a70?, 0xc00365e540}, 0xc003680900)
 /go/pkg/mod/github.com/opentracing-contrib/[email protected]/nethttp/server.go:154 +0x623
net/http.HandlerFunc.ServeHTTP(0x0?, {0x5a25a70?, 0xc00365e540?}, 0xc000092c00?)
 /usr/local/go/src/net/http/server.go:2084 +0x2f
net/http.serverHandler.ServeHTTP({0xc0036dc120?}, {0x5a25a70, 0xc00365e540}, 0xc003680900)
 /usr/local/go/src/net/http/server.go:2916 +0x43b
net/http.(*conn).serve(0xc003625ea0, {0x5a28838, 0xc000c73ef0})
 /usr/local/go/src/net/http/server.go:1966 +0x5d7
created by net/http.(*Server).Serve
 /usr/local/go/src/net/http/server.go:3071 +0x4db

@quodlibetor thank you for reporting this issue. We have been able to reproduce it. It seems related to a known issue in loki's promtail when fetching target details. See https://github.com/grafana/loki/issues/6930. Please hold on until the issue has been fixed upstream and we can release a new version

The underlying issue in loki was solved via grafana/loki#7771 which is included in the updated version we depend on from #2590.

Closing as completed.