pushgateway icon indicating copy to clipboard operation
pushgateway copied to clipboard

update with pushgateway updated libs

Open casmssm opened this issue 3 years ago • 1 comments

Hi, recently We got this error "fatal error: concurrent map iteration and map write".

I work at Globo.com and we tested the new compiled version of your modified pushgateway (TTL) with new libs from pushgateway. It's working until now.

Then I sent this pull request.

Thanks for your contribution.

casmssm avatar Jul 11 '22 17:07 casmssm

Hi @casmssm have you solved the problems ? Now,i using your branch fix/fatal_error_concurrent_map_iteration_and_map_write,and met the same problem. But i don't konw how to fix.

the flow is part of log stack

ts=2023-02-28T07:41:44.787Z caller=main.go:100 level=info msg="starting pushgateway" version="(version=1.5.1, branch=HEAD, revision=7afc96cfc5e2b278f3b20e56968ff30eea22b70e)"
ts=2023-02-28T07:41:44.788Z caller=main.go:101 level=info build_context="(go=go1.19.3, user=root@fc81889ee1a6, date=20221129-16:30:38)"
ts=2023-02-28T07:41:44.791Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9091
ts=2023-02-28T07:41:44.791Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9091
level=info ts=2023-03-07T09:54:11.778Z caller=main.go:86 msg="starting pushgateway" version="(version=, branch=, revision=)"
level=info ts=2023-03-07T09:54:11.778Z caller=main.go:87 build_context="(go=go1.20.1, user=, date=)"
level=info ts=2023-03-07T09:54:11.783Z caller=main.go:140 listen_address=:9091
level=info ts=2023-03-07T09:54:11.783Z caller=tls_config.go:191 msg="TLS is disabled." http2=false
fatal error: concurrent map iteration and map write

goroutine 127 [running]:
github.com/prometheus/pushgateway/storage.(*DiskMetricStore).GetMetricFamiliesMap(0xc0001aa840)
	/home/mengh/pgw/pushgateway-ttl-1.4.0/storage/diskmetricstore.go:199 +0x24c
github.com/prometheus/pushgateway/storage.(*DiskMetricStore).checkWriteRequest(0xc0001aa840, {0xc000dc9bf0, {0xc0f9f4c406a2e169, 0x130b8ded8bf7, 0x1023080}, 0xc000dc9c50, 0x1, 0xc001df4cc0})
	/home/mengh/pgw/pushgateway-ttl-1.4.0/storage/diskmetricstore.go:377 +0x259
github.com/prometheus/pushgateway/storage.(*DiskMetricStore).loop(0xc0001aa840, 0x45d964b800)
	/home/mengh/pgw/pushgateway-ttl-1.4.0/storage/diskmetricstore.go:235 +0x2dd
created by github.com/prometheus/pushgateway/storage.NewDiskMetricStore
	/home/mengh/pgw/pushgateway-ttl-1.4.0/storage/diskmetricstore.go:104 +0x4ac

goroutine 1 [IO wait]:
internal/poll.runtime_pollWait(0x7f11e125a018, 0x72)
	/home/mengh/pgw/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc000160180?, 0x8?, 0x0)
	/home/mengh/pgw/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/home/mengh/pgw/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000160180)
	/home/mengh/pgw/go/src/internal/poll/fd_unix.go:614 +0x2bd
net.(*netFD).accept(0xc000160180)
	/home/mengh/pgw/go/src/net/fd_unix.go:172 +0x35
net.(*TCPListener).accept(0xc000013f20)
	/home/mengh/pgw/go/src/net/tcpsock_posix.go:148 +0x25
net.(*TCPListener).Accept(0xc000013f20)
	/home/mengh/pgw/go/src/net/tcpsock.go:297 +0x3d
net/http.(*Server).Serve(0xc0002631d0, {0xc8baf0, 0xc000013f20})
	/home/mengh/pgw/go/src/net/http/server.go:3059 +0x385
github.com/prometheus/exporter-toolkit/web.Serve({0xc8baf0, 0xc000013f20}, 0xc0002631d0, {0x0?, 0xc00076a300?}, {0xc87fe0?, 0xc000276a50?})
	/root/go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:192 +0x72d
main.main()
	/home/mengh/pgw/pushgateway-ttl-1.4.0/main.go:199 +0x343d

menghe999 avatar Mar 14 '23 12:03 menghe999