opentelemetry-collector-contrib icon indicating copy to clipboard operation
opentelemetry-collector-contrib copied to clipboard

[exporter/elasticsearch] goroutine leaks in tests

Open mauri870 opened this issue 1 year ago • 5 comments

Component(s)

exporter/elasticsearch

Describe the issue you're reporting

Description

The goleak tool reports goroutine leaks in tests.

Steps to Reproduce

cd exporter/elasticsearchexporter
go test ./... -v

Expected Result

No leaks reported by goleak.

Actual Result

goleak: Errors on successful test run: found unexpected goroutines:
[Goroutine 765 in state select, with net/http.(*persistConn).writeLoop on top of the stack:
net/http.(*persistConn).writeLoop(0xc000485320)
        /home/mauri870/gopath/pkg/mod/golang.org/[email protected]/src/net/http/transport.go:2519 +0xe7
created by net/http.(*Transport).dialConn in goroutine 823
        /home/mauri870/gopath/pkg/mod/golang.org/[email protected]/src/net/http/transport.go:1875 +0x15a5
 Goroutine 764 in state IO wait, with internal/poll.runtime_pollWait on top of the stack:
internal/poll.runtime_pollWait(0x70ffb101fb60, 0x72)
        /home/mauri870/gopath/pkg/mod/golang.org/[email protected]/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc000418d80?, 0xc000590000?, 0x0)
        /home/mauri870/gopath/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /home/mauri870/gopath/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000418d80, {0xc000590000, 0x1000, 0x1000})

[...]

Related issues

  • https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/30438

mauri870 avatar Oct 07 '24 11:10 mauri870

Pinging code owners:

  • exporter/elasticsearch: @JaredTan95 @carsonip @lahsivjar

See Adding Labels via Comments if you do not have permissions to add labels yourself.

github-actions[bot] avatar Oct 07 '24 11:10 github-actions[bot]

As of now there are two leaks that are detected, following the goleak docs I was able to pinpoint the exact tests that have leaks:

$ export GOTOOLCHAIN=go1.23.1
$ export GOSUMDB=sum.golang.org

$ cd exporter/elasticsearchexporter
$ go test -c -o tests; for test in $(go test -list . | grep -E "^(Test|Example)"); do ./tests -test.run "^$test\$" &>/dev/null && echo -n "." || echo -e "\n$test failed"; done

.......................
TestComponentLifecycle failed
................%

I was unable to fix this particular test, but commenting it reports TestExporterMetrics instead, which I fixed at https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/35639. After that no more leaks are reported.

I would appreciate it if someone could take a look at TestComponentLifecycle.

mauri870 avatar Oct 07 '24 11:10 mauri870

/label -needs-triage

carsonip avatar Oct 07 '24 13:10 carsonip

Turns out TestExporterMetrics was being reported because of a flaky assertion. We should probably fix TestComponentLifecycle first and make sure all the other tests are not being reported by goleak.

mauri870 avatar Oct 07 '24 17:10 mauri870

I have found another leak of unclosed bulk indexer, fixing in https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/35865/commits/210306ef6885f6f4fb88b3b31f759397acfae2fb as part of #35865

carsonip avatar Oct 18 '24 08:10 carsonip

Is this still relevant? I'm not able to reproduce the failure locally.

axw avatar Nov 21 '24 03:11 axw

Can no longer reproduce any leaks. This issue should be good to close.

carsonip avatar Nov 21 '24 14:11 carsonip

Can no longer reproduce any leaks. This issue should be good to close.

I'm gonna close this for now then. Feel free to re-open if there is still evidence that it's not fixed.

ChrsMark avatar Nov 21 '24 16:11 ChrsMark