dfuse-eosio icon indicating copy to clipboard operation
dfuse-eosio copied to clipboard

Published 20 hours ago • verified publisher icon dfuse-io

Mindreader `sudog with non-nil elem` with non-nil elem

Open maoueh opened this issue 4 years ago • 2 comments

EOS Mainnet mindreader panicked with fatal error: runtime: sudog with non-nil elem. This comes from dfuse-eosio/codec/consolereader.go:109:

File dfuse-eosio/codec/consolereader.go
106: func (l *ConsoleReader) Read() (out interface{}, err error) {
107: 	ctx := l.ctx
108: 
109: 	for line := range l.readBuffer {
110: 		line = line[6:]
111:
Stacktrace 
fatal error: runtime: sudog with non-nil elem

goroutine 165 [running]:
runtime.throw(0x1e53d78, 0x20)
	/usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc00fb71c68 sp=0xc00fb71c38 pc=0x440282
runtime.releaseSudog(0xc00e5ba300)
	/usr/local/go/src/runtime/proc.go:360 +0x387 fp=0xc00fb71cd8 sp=0xc00fb71c68 pc=0x4434c7
runtime.chanrecv(0xc000096fc0, 0xc000749e20, 0xc000749e01, 0xc000000101)
	/usr/local/go/src/runtime/chan.go:539 +0x35b fp=0xc00fb71d68 sp=0xc00fb71cd8 pc=0x4115db
runtime.chanrecv2(0xc000096fc0, 0xc000749e20, 0x99)
	/usr/local/go/src/runtime/chan.go:412 +0x2b fp=0xc00fb71d98 sp=0xc00fb71d68 pc=0x41126b
github.com/dfuse-io/dfuse-eosio/codec.(*ConsoleReader).Read(0xc000213f40, 0xc000749e90, 0x40, 0x0, 0xed652f152)
	/work/src/codec/consolereader.go:109 +0x88 fp=0xc00fb71e50 sp=0xc00fb71d98 pc=0xe68a38
github.com/dfuse-io/manageos/mindreader.(*MindReaderPlugin).readOneMessage(0xc0003683f0, 0xc00074c0c0, 0x0, 0x0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/mindreader/mindreader.go:268 +0x38 fp=0xc00fb71eb8 sp=0xc00fb71e50 pc=0x15c3a98
github.com/dfuse-io/manageos/mindreader.(*MindReaderPlugin).ReadFlow(0xc0003683f0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/mindreader/mindreader.go:187 +0xc7 fp=0xc00fb71fd8 sp=0xc00fb71eb8 pc=0x15c27b7
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc00fb71fe0 sp=0xc00fb71fd8 pc=0x473381
created by github.com/dfuse-io/manageos/mindreader.RunMindReaderPlugin
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/mindreader/mindreader.go:136 +0x69e

goroutine 1 [select, 8293 minutes]:
github.com/dfuse-io/dfuse-eosio/launcher/cli.dfuseStartE(0x46b1440, 0xc0001adb00, 0x1, 0x23, 0x0, 0x0)
	/work/src/launcher/cli/start.go:117 +0xbbc
github.com/spf13/cobra.(*Command).execute(0x46b1440, 0xc0001ad200, 0x23, 0x24, 0x46b1440, 0xc0001ad200)
	/work/go/pkg/mod/github.com/spf13/[email protected]/command.go:838 +0x453
github.com/spf13/cobra.(*Command).ExecuteC(0x46b0f00, 0x2d6, 0xc00059fef8, 0x1)
	/work/go/pkg/mod/github.com/spf13/[email protected]/command.go:943 +0x317
github.com/spf13/cobra.(*Command).Execute(...)
	/work/go/pkg/mod/github.com/spf13/[email protected]/command.go:883
github.com/dfuse-io/dfuse-eosio/launcher/cli.Main()
	/work/src/launcher/cli/main.go:61 +0x6b5
main.main()
	/work/src/cmd/dfuseeos/main.go:13 +0x20

goroutine 36 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc00012a870)
	/work/go/pkg/mod/[email protected]/stats/view/worker.go:154 +0x100
created by go.opencensus.io/stats/view.init.0
	/work/go/pkg/mod/[email protected]/stats/view/worker.go:32 +0x57

goroutine 12 [select, 8293 minutes]:
github.com/desertbit/timer.timerRoutine()
	/work/go/pkg/mod/github.com/desertbit/[email protected]/timers.go:119 +0x304
created by github.com/desertbit/timer.init.0
	/work/go/pkg/mod/github.com/desertbit/[email protected]/timers.go:15 +0x35

goroutine 13 [select, 8293 minutes]:
github.com/blevesearch/bleve/index.AnalysisWorker(0xc0002af620, 0xc0002af680)
	/work/go/pkg/mod/github.com/fproulx-eoscanada/[email protected]/index/analysis.go:102 +0x102
created by github.com/blevesearch/bleve/index.NewAnalysisQueue
	/work/go/pkg/mod/github.com/fproulx-eoscanada/[email protected]/index/analysis.go:94 +0xc8

goroutine 14 [select, 8293 minutes]:
github.com/blevesearch/bleve/index.AnalysisWorker(0xc0002af620, 0xc0002af680)
	/work/go/pkg/mod/github.com/fproulx-eoscanada/[email protected]/index/analysis.go:102 +0x102
created by github.com/blevesearch/bleve/index.NewAnalysisQueue
	/work/go/pkg/mod/github.com/fproulx-eoscanada/[email protected]/index/analysis.go:94 +0xc8

goroutine 15 [select, 8293 minutes]:
github.com/blevesearch/bleve/index.AnalysisWorker(0xc0002af620, 0xc0002af680)
	/work/go/pkg/mod/github.com/fproulx-eoscanada/[email protected]/index/analysis.go:102 +0x102
created by github.com/blevesearch/bleve/index.NewAnalysisQueue
	/work/go/pkg/mod/github.com/fproulx-eoscanada/[email protected]/index/analysis.go:94 +0xc8

goroutine 16 [select, 8293 minutes]:
github.com/blevesearch/bleve/index.AnalysisWorker(0xc0002af620, 0xc0002af680)
	/work/go/pkg/mod/github.com/fproulx-eoscanada/[email protected]/index/analysis.go:102 +0x102
created by github.com/blevesearch/bleve/index.NewAnalysisQueue
	/work/go/pkg/mod/github.com/fproulx-eoscanada/[email protected]/index/analysis.go:94 +0xc8

goroutine 439 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0004aa190, 0x1, 0x0, 0x0, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:395 +0x122
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000532180, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:513 +0x1cd
google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc0006d5200)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:296 +0xcb
created by google.golang.org/grpc/internal/transport.newHTTP2Server
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:293 +0x10c1

goroutine 64 [select, 1 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000213640)
	/work/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:69 +0xc2
created by google.golang.org/grpc.newCCBalancerWrapper
	/work/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:60 +0x16d

goroutine 95 [select]:
github.com/dfuse-io/manageos/mindreader.(*MindReaderPlugin).consumeReadFlow(0xc0003683f0, 0xc00074c0c0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/mindreader/mindreader.go:224 +0x10c
created by github.com/dfuse-io/manageos/mindreader.(*MindReaderPlugin).ReadFlow
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/mindreader/mindreader.go:182 +0x87

goroutine 65 [chan receive, 1 minutes]:
google.golang.org/grpc.(*addrConn).resetTransport(0xc0001efb80)
	/work/go/pkg/mod/google.golang.org/[email protected]/clientconn.go:1164 +0x6ea
created by google.golang.org/grpc.(*addrConn).connect
	/work/go/pkg/mod/google.golang.org/[email protected]/clientconn.go:800 +0x128

goroutine 98 [select, 31 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0002138c0)
	/work/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:69 +0xc2
created by google.golang.org/grpc.newCCBalancerWrapper
	/work/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:60 +0x16d

goroutine 99 [chan receive, 31 minutes]:
google.golang.org/grpc.(*addrConn).resetTransport(0xc0004a5080)
	/work/go/pkg/mod/google.golang.org/[email protected]/clientconn.go:1164 +0x6ea
created by google.golang.org/grpc.(*addrConn).connect
	/work/go/pkg/mod/google.golang.org/[email protected]/clientconn.go:800 +0x128

goroutine 100 [IO wait, 8292 minutes]:
internal/poll.runtime_pollWait(0x7f39ff554fa0, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000696498, 0x72, 0x0, 0x0, 0x1e1e9a3)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000696480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1d4
net.(*netFD).accept(0xc000696480, 0xafbf2ce2899afd01, 0x1000000000000, 0xafbf2ce2899afd01)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc000190560, 0x5eb962c2, 0xc000033dd8, 0x4d7a76)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc000190560, 0xc000033e28, 0x18, 0xc000103e00, 0x782b3c)
	/usr/local/go/src/net/tcpsock.go:261 +0x64
net/http.(*Server).Serve(0xc0000b6000, 0x370ea40, 0xc000190560, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2901 +0x25d
net/http.(*Server).ListenAndServe(0xc0000b6000, 0xc0004a7fb0, 0x0)
	/usr/local/go/src/net/http/server.go:2830 +0xb7
github.com/dfuse-io/dmetrics.Serve(0x1e1b6ab, 0x5)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/http.go:25 +0xab
created by github.com/dfuse-io/dfuse-eosio/launcher/cli.setup
	/work/src/launcher/cli/setup.go:36 +0x66

goroutine 101 [IO wait, 8293 minutes]:
internal/poll.runtime_pollWait(0x7f39ff554ec0, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc00011eb18, 0x72, 0x0, 0x0, 0x1e1e9a3)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc00011eb00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1d4
net.(*netFD).accept(0xc00011eb00, 0xc00007bc78, 0xc000437880, 0x7f39ff6cce98)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc000384440, 0xc00007bcb8, 0x418738, 0x30)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc000384440, 0x1c8e7c0, 0xc000798840, 0x1ad3620, 0x469d990)
	/usr/local/go/src/net/tcpsock.go:261 +0x64
net/http.(*Server).Serve(0xc0004d8000, 0x370ea40, 0xc000384440, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2901 +0x25d
net/http.(*Server).ListenAndServe(0xc0004d8000, 0xc0004d8000, 0x0)
	/usr/local/go/src/net/http/server.go:2830 +0xb7
net/http.ListenAndServe(...)
	/usr/local/go/src/net/http/server.go:3086
github.com/dfuse-io/dfuse-eosio/launcher/cli.setup.func1()
	/work/src/launcher/cli/setup.go:45 +0x69
created by github.com/dfuse-io/dfuse-eosio/launcher/cli.setup
	/work/src/launcher/cli/setup.go:43 +0x211

goroutine 20883 [runnable]:
internal/poll.runtime_pollWait(0x7f39ff555400, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc005394318, 0x72, 0x8001, 0x8000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc005394300, 0xc005980000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
os.(*File).read(...)
	/usr/local/go/src/os/file_unix.go:263
os.(*File).Read(0xc011ad5778, 0xc005980000, 0x8000, 0x8000, 0xe8, 0x0, 0x0)
	/usr/local/go/src/os/file.go:116 +0x71
io.copyBuffer(0x36d2140, 0xc0056ec390, 0x36d66e0, 0xc011ad5778, 0xc005980000, 0x8000, 0x8000, 0xc0180f27a0, 0x41115f, 0xc005248cb8)
	/usr/local/go/src/io/io.go:405 +0x122
io.Copy(...)
	/usr/local/go/src/io/io.go:364
os/exec.(*Cmd).writerDescriptor.func1(0xc0180f27b8, 0x7d87e7)
	/usr/local/go/src/os/exec/exec.go:311 +0x63
os/exec.(*Cmd).Start.func1(0xc005367340, 0xc00d968be0)
	/usr/local/go/src/os/exec/exec.go:441 +0x27
created by os/exec.(*Cmd).Start
	/usr/local/go/src/os/exec/exec.go:440 +0x632

goroutine 440 [select]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0006d5200)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:968 +0x285
created by google.golang.org/grpc/internal/transport.newHTTP2Server
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:302 +0x10e6

goroutine 441 [IO wait]:
internal/poll.runtime_pollWait(0x7f39ff5548a0, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000696218, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000696200, 0xc00060c000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc000696200, 0xc00060c000, 0x8000, 0x8000, 0x203003, 0x0, 0x60100000008)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000118018, 0xc00060c000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
bufio.(*Reader).Read(0xc0002ce300, 0xc00067e3b8, 0x9, 0x9, 0xc00a659880, 0x1007f39ff6cc108, 0x0)
	/usr/local/go/src/bufio/bufio.go:226 +0x24f
io.ReadAtLeast(0x36d1280, 0xc0002ce300, 0xc00067e3b8, 0x9, 0x9, 0x9, 0x5ec0fa52, 0xc00532adc8, 0x4d7a76)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc00067e3b8, 0x9, 0x9, 0x36d1280, 0xc0002ce300, 0x0, 0x0, 0x0, 0x0)
	/work/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00067e380, 0xc0091cfb60, 0x47782a0, 0x0, 0x0)
	/work/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0006d5200, 0xc0004a01b0, 0x3392dc0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:458 +0x98
google.golang.org/grpc.(*Server).serveStreams(0xc00020e000, 0x372cce0, 0xc0006d5200)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:718 +0xdb
google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00020e000, 0x372cce0, 0xc0006d5200)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:679 +0x3f
created by google.golang.org/grpc.(*Server).handleRawConn
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:678 +0x562

goroutine 428 [IO wait]:
internal/poll.runtime_pollWait(0x7f39ff554de0, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000660a98, 0x72, 0x0, 0x0, 0x1e1e9a3)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000660a80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1d4
net.(*netFD).accept(0xc000660a80, 0x71276b40264ab14d, 0x0, 0x71276b40264ab14d)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc0005e5be0, 0x5ec0fa50, 0xc000079de0, 0x4d7a76)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc0005e5be0, 0xc000079e30, 0x18, 0xc000022300, 0x782b3c)
	/usr/local/go/src/net/tcpsock.go:261 +0x64
net/http.(*Server).Serve(0xc0000b70a0, 0x370ea40, 0xc0005e5be0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2901 +0x25d
net/http.(*Server).ListenAndServe(0xc0000b70a0, 0x0, 0x473381)
	/usr/local/go/src/net/http/server.go:2830 +0xb7
github.com/dfuse-io/manageos/operator.(*Operator).RunHTTPServer.func2(0xc0000b70a0, 0xc0000969c0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/operator/http_server.go:81 +0x2f
created by github.com/dfuse-io/manageos/operator.(*Operator).RunHTTPServer
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/operator/http_server.go:80 +0x1213

goroutine 103 [select, 8293 minutes]:
github.com/dfuse-io/dfuse-eosio/launcher.(*Launcher).Launch.func3(0xc000200880, 0x1e23b19, 0xa, 0x371ba40, 0xc0002c7020)
	/work/src/launcher/launcher.go:123 +0xd3
created by github.com/dfuse-io/dfuse-eosio/launcher.(*Launcher).Launch
	/work/src/launcher/launcher.go:122 +0x5b5

goroutine 104 [sleep]:
time.Sleep(0x12a05f200)
	/usr/local/go/src/runtime/time.go:188 +0xba
github.com/dfuse-io/dfuse-eosio/launcher.(*Launcher).Launch.func4(0xc000200880)
	/work/src/launcher/launcher.go:146 +0x45
created by github.com/dfuse-io/dfuse-eosio/launcher.(*Launcher).Launch
	/work/src/launcher/launcher.go:142 +0x647

goroutine 105 [sleep]:
time.Sleep(0x12a05f200)
	/usr/local/go/src/runtime/time.go:188 +0xba
github.com/dfuse-io/dfuse-eosio/metrics.(*Manager).Launch(0xc0007aaa80, 0x1)
	/work/src/metrics/manager.go:146 +0x4f2
created by github.com/dfuse-io/dfuse-eosio/launcher/cli.dfuseStartE
	/work/src/launcher/cli/start.go:112 +0xb23

goroutine 106 [syscall, 8293 minutes]:
os/signal.signal_recv(0x1)
	/usr/local/go/src/runtime/sigqueue.go:147 +0x9c
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.Notify.func1
	/usr/local/go/src/os/signal/signal.go:127 +0x44

goroutine 108 [chan receive, 8293 minutes]:
github.com/dfuse-io/derr.SetupSignalHandler.func1(0xc0000a2d20, 0xc0004f8810, 0x0, 0xc0007aaba0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/signals.go:48 +0x55
created by github.com/dfuse-io/derr.SetupSignalHandler
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/signals.go:46 +0x112

goroutine 111 [IO wait]:
internal/poll.runtime_pollWait(0x7f39ff555080, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000200b18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000200b00, 0xc000095000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc000200b00, 0xc000095000, 0x1000, 0x1000, 0x4, 0xc00a61f4c0, 0x4)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00019a420, 0xc000095000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
net/http.(*connReader).Read(0xc0005fb2c0, 0xc000095000, 0x1000, 0x1000, 0x4f1944, 0xc015da17d0, 0x203003)
	/usr/local/go/src/net/http/server.go:786 +0xf4
bufio.(*Reader).fill(0xc0007aad80)
	/usr/local/go/src/bufio/bufio.go:100 +0x103
bufio.(*Reader).ReadSlice(0xc0007aad80, 0xa, 0x28, 0xc0002bd9c0, 0x417ec6, 0xc00d279b00, 0x100)
	/usr/local/go/src/bufio/bufio.go:359 +0x3d
bufio.(*Reader).ReadLine(0xc0007aad80, 0xc0002bd9c8, 0xc00a659880, 0x7f39ff6cce98, 0x0, 0x418738, 0x30)
	/usr/local/go/src/bufio/bufio.go:388 +0x34
net/textproto.(*Reader).readLineSlice(0xc00d0b5f50, 0xc00d279b00, 0x0, 0x442d9c, 0xc0002bda28, 0x0)
	/usr/local/go/src/net/textproto/reader.go:58 +0x6c
net/textproto.(*Reader).ReadLine(...)
	/usr/local/go/src/net/textproto/reader.go:39
net/http.readRequest(0xc0007aad80, 0x0, 0xc00d279b00, 0x0, 0x0)
	/usr/local/go/src/net/http/request.go:1015 +0xa4
net/http.(*conn).readRequest(0xc00062a500, 0x3716080, 0xc000213c00, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:966 +0x191
net/http.(*conn).serve(0xc00062a500, 0x3716080, 0xc000213c00)
	/usr/local/go/src/net/http/server.go:1822 +0x6d4
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2933 +0x35c

goroutine 109 [IO wait]:
internal/poll.runtime_pollWait(0x7f39ff555160, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc00011f098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00011f080, 0xc00002b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc00011f080, 0xc00002b000, 0x1000, 0x1000, 0x442d9c, 0xc000657b70, 0x46fd30)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00019a418, 0xc00002b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
net/http.(*persistConn).Read(0xc0004e6240, 0xc00002b000, 0x1000, 0x1000, 0xc0000a2f60, 0xc000657c70, 0x410715)
	/usr/local/go/src/net/http/transport.go:1825 +0x75
bufio.(*Reader).fill(0xc0007aac60)
	/usr/local/go/src/bufio/bufio.go:100 +0x103
bufio.(*Reader).Peek(0xc0007aac60, 0x1, 0x0, 0x0, 0x1, 0xc00d164000, 0x0)
	/usr/local/go/src/bufio/bufio.go:138 +0x4f
net/http.(*persistConn).readLoop(0xc0004e6240)
	/usr/local/go/src/net/http/transport.go:1978 +0x1a8
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1647 +0xc56

goroutine 110 [select]:
net/http.(*persistConn).writeLoop(0xc0004e6240)
	/usr/local/go/src/net/http/transport.go:2277 +0x11c
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1648 +0xc7b

goroutine 164 [runnable]:
bufio.(*Scanner).Scan(0xc00011fb80, 0xc00047ffb8)
	/usr/local/go/src/bufio/scan.go:134 +0x9c3
github.com/dfuse-io/dfuse-eosio/codec.(*ConsoleReader).setupScanner.func1(0xc000213f40)
	/work/src/codec/consolereader.go:68 +0x33
created by github.com/dfuse-io/dfuse-eosio/codec.(*ConsoleReader).setupScanner
	/work/src/codec/consolereader.go:67 +0x141

goroutine 427 [select, 8284 minutes]:
github.com/dfuse-io/manageos/operator.(*Operator).Launch(0xc0000969c0, 0x1, 0x7fff97122992, 0x5, 0xc000010620, 0x1, 0x1, 0x0, 0x0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/operator/operator.go:139 +0x405
created by github.com/dfuse-io/manageos/app/nodeos_mindreader.(*App).Run
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/app/nodeos_mindreader/app.go:223 +0x1183

goroutine 96 [semacquire]:
sync.runtime_Semacquire(0xc00029ea90)
	/usr/local/go/src/runtime/sema.go:56 +0x42
sync.(*WaitGroup).Wait(0xc00029ea88)
	/usr/local/go/src/sync/waitgroup.go:130 +0x64
golang.org/x/sync/errgroup.(*Group).Wait(0xc00029ea80, 0xc00f65cf80, 0x4)
	/work/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:40 +0x31
github.com/dfuse-io/manageos/mindreader.(*DefaultArchiver).uploadFiles(0xc000213e80, 0xc003e58e00, 0x2)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/mindreader/archiver.go:149 +0x344
github.com/dfuse-io/manageos/mindreader.(*MindReaderPlugin).alwaysUploadFiles(0xc0003683f0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/mindreader/mindreader.go:208 +0x6b
created by github.com/dfuse-io/manageos/mindreader.(*MindReaderPlugin).ReadFlow
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/mindreader/mindreader.go:183 +0xac

goroutine 453 [select]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0004be000)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:968 +0x285
created by google.golang.org/grpc/internal/transport.newHTTP2Server
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:302 +0x10e6

goroutine 166 [IO wait, 8293 minutes]:
internal/poll.runtime_pollWait(0x7f39ff555320, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc00011fd98, 0x72, 0x0, 0x0, 0x1e1e9a3)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc00011fd80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1d4
net.(*netFD).accept(0xc00011fd80, 0x37e11d600, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc000111c20, 0xc000032e48, 0xc000032e50, 0x18)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc000111c20, 0x3392d88, 0xc00020e000, 0x372aa00, 0xc00019a258)
	/usr/local/go/src/net/tcpsock.go:261 +0x64
google.golang.org/grpc.(*Server).Serve(0xc00020e000, 0x370ea40, 0xc000111c20, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:597 +0x210
github.com/dfuse-io/manageos/mindreader.RunGRPCServer.func1(0xc00020e000, 0x370ea40, 0xc000111c20, 0xc000097020)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/mindreader/publisher.go:37 +0x43
created by github.com/dfuse-io/manageos/mindreader.RunGRPCServer
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/mindreader/publisher.go:36 +0x2e8

goroutine 452 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00012a0a0, 0x1, 0x0, 0x0, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:395 +0x122
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0002cf380, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:513 +0x1cd
google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc0004be000)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:296 +0xcb
created by google.golang.org/grpc/internal/transport.newHTTP2Server
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:293 +0x10c1

goroutine 7345 [select]:
github.com/dfuse-io/dmetrics.(*HeadTimeDrift).SetBlockTime.func1(0xc000110660)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/head_time_drift.go:60 +0x1da
created by github.com/dfuse-io/dmetrics.(*HeadTimeDrift).SetBlockTime
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/head_time_drift.go:57 +0x8c

goroutine 454 [IO wait]:
internal/poll.runtime_pollWait(0x7f39ff5547c0, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc00011e118, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00011e100, 0xc00079a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc00011e100, 0xc00079a000, 0x8000, 0x8000, 0x800010601, 0x0, 0x1060100000000)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0000100e8, 0xc00079a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
bufio.(*Reader).Read(0xc000128360, 0xc0000b68f8, 0x9, 0x9, 0xc00ab21180, 0x7f39ff6cc7d0, 0x0)
	/usr/local/go/src/bufio/bufio.go:226 +0x24f
io.ReadAtLeast(0x36d1280, 0xc000128360, 0xc0000b68f8, 0x9, 0x9, 0x9, 0x9436d5, 0xc00515d72c, 0xc01191e1c0)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc0000b68f8, 0x9, 0x9, 0x36d1280, 0xc000128360, 0x0, 0x0, 0xc00515d720, 0x0)
	/work/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0000b68c0, 0xc00515d720, 0x47782a0, 0x0, 0x0)
	/work/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0004be000, 0xc0005dddd0, 0x3392dc0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:458 +0x98
google.golang.org/grpc.(*Server).serveStreams(0xc00020e000, 0x372cce0, 0xc0004be000)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:718 +0xdb
google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00020e000, 0x372cce0, 0xc0004be000)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:679 +0x3f
created by google.golang.org/grpc.(*Server).handleRawConn
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:678 +0x562

goroutine 20854 [select]:
io.(*pipe).Write(0xc00074c060, 0xc016f9c500, 0xa1, 0x140, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/pipe.go:94 +0x1db
io.(*PipeWriter).Write(...)
	/usr/local/go/src/io/pipe.go:163
github.com/dfuse-io/manageos/mindreader.(*MindReaderPlugin).LogLine(0xc0003683f0, 0xc016a60a00, 0xa0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/mindreader/mindreader.go:300 +0xaa
github.com/dfuse-io/manageos/superviser.(*Superviser).processLogLine(0xc00066a120, 0xc016a60a00, 0xa0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/superviser/superviser.go:185 +0xb7
github.com/dfuse-io/manageos/superviser.(*Superviser).start(0xc00066a120, 0xc005398c60)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/superviser/superviser.go:173 +0x1e4
created by github.com/dfuse-io/manageos/superviser.(*Superviser).Start
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/superviser/superviser.go:111 +0x48e

goroutine 474 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00060a410, 0x1, 0x0, 0x0, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:395 +0x122
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0005f2480, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:513 +0x1cd
google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc003ef8600)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:296 +0xcb
created by google.golang.org/grpc/internal/transport.newHTTP2Server
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:293 +0x10c1

goroutine 426 [select]:
github.com/dfuse-io/manageos.(*MetricsAndReadinessManager).Launch(0xc000552210)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/monitor.go:54 +0xdc
created by github.com/dfuse-io/manageos/app/nodeos_mindreader.(*App).Run
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/app/nodeos_mindreader/app.go:222 +0x1112

goroutine 433 [select]:
github.com/dfuse-io/bstream/blockstream.(*Server).Blocks(0xc0006145a0, 0xc0000a0870, 0x3726200, 0xc0057ffed0, 0x0, 0x0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/blockstream/server.go:95 +0x420
github.com/dfuse-io/pbgo/dfuse/bstream/v1._BlockStream_Blocks_Handler(0x1c87ec0, 0xc0006145a0, 0x37232c0, 0xc00045d5e0, 0xc00068bcb0, 0x6)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/dfuse/bstream/v1/bstream.pb.go:462 +0x109
github.com/dfuse-io/dgrpc.setupLoggingInterceptors.func2(0x1c87ec0, 0xc0006145a0, 0x37232c0, 0xc00045d5e0, 0xc00045d440, 0x33904e0, 0x0, 0x0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/logging.go:68 +0xd1
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x1c87ec0, 0xc0006145a0, 0x37232c0, 0xc00045d5e0, 0x3716140, 0xc00068bbc0)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x5f
github.com/dfuse-io/dgrpc.setupTracingInterceptors.func2(0x1c87ec0, 0xc0006145a0, 0x37232c0, 0xc00045d5e0, 0xc00045d440, 0xc00045d460, 0x3716101, 0xc00045d5e0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/tracing.go:39 +0xe8
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x1c87ec0, 0xc0006145a0, 0x37232c0, 0xc00045d5e0, 0x24, 0xbfa67689b876a45e)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x5f
github.com/grpc-ecosystem/go-grpc-middleware/logging/zap.StreamServerInterceptor.func1(0x1c87ec0, 0xc0006145a0, 0x3723440, 0xc00045d580, 0xc00045d440, 0xc00045d480, 0xc000318901, 0xc00045d580)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/logging/zap/server_interceptors.go:59 +0x16a
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x1c87ec0, 0xc0006145a0, 0x3723440, 0xc00045d580, 0x24, 0xc0000a0820)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x5f
github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).StreamServerInterceptor.func1(0x1c87ec0, 0xc0006145a0, 0x3723320, 0xc000318940, 0xc00045d440, 0xc00045d4a0, 0xc000010601, 0xc000318940)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/server_metrics.go:121 +0xeb
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x1c87ec0, 0xc0006145a0, 0x3723320, 0xc000318940, 0x7f39ff6cd560, 0x0)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x5f
github.com/grpc-ecosystem/go-grpc-middleware/tags.StreamServerInterceptor.func1(0x1c87ec0, 0xc0006145a0, 0x3723860, 0xc00051e180, 0xc00045d440, 0xc00045d4c0, 0xc00045d520, 0xc00045d440)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/tags/interceptors.go:38 +0x265
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x1c87ec0, 0xc0006145a0, 0x3723860, 0xc00051e180, 0xc00007ac50, 0x418738)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x5f
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1(0x1c87ec0, 0xc0006145a0, 0x3723860, 0xc00051e180, 0xc00045d440, 0x33904e0, 0x3716140, 0xc00068b8c0)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:58 +0xcf
google.golang.org/grpc.(*Server).processStreamingRPC(0xc00020e000, 0x372cce0, 0xc000023200, 0xc0006d3300, 0xc000552b10, 0x46a2440, 0x0, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:1244 +0x505
google.golang.org/grpc.(*Server).handleStream(0xc00020e000, 0x372cce0, 0xc000023200, 0xc0006d3300, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:1317 +0xcd6
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000298634, 0xc00020e000, 0x372cce0, 0xc000023200, 0xc0006d3300)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:722 +0xa1
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:720 +0xa1

goroutine 5479217229 [IO wait]:
internal/poll.runtime_pollWait(0x7f39ff554980, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc004350f18, 0x72, 0x1000, 0x1001, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc004350f00, 0xc009a16000, 0x1001, 0x1001, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc004350f00, 0xc009a16000, 0x1001, 0x1001, 0x203004, 0x6b7480, 0xc00d295638)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0116b0e58, 0xc009a16000, 0x1001, 0x1001, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
crypto/tls.(*atLeastReader).Read(0xc010341ee0, 0xc009a16000, 0x1001, 0x1001, 0x1e, 0xffc, 0xc00532e9c8)
	/usr/local/go/src/crypto/tls/conn.go:760 +0x60
bytes.(*Buffer).ReadFrom(0xc00d295758, 0x36d1540, 0xc010341ee0, 0x415e15, 0x1b3b2a0, 0x1d9fb40)
	/usr/local/go/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc00d295500, 0x36d62a0, 0xc0116b0e58, 0x5, 0xc0116b0e58, 0xd)
	/usr/local/go/src/crypto/tls/conn.go:782 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc00d295500, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:589 +0x115
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:557
crypto/tls.(*Conn).Read(0xc00d295500, 0xc010061000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1233 +0x15b
bufio.(*Reader).Read(0xc00e465ec0, 0xc0054b38b8, 0x9, 0x9, 0xc010061009, 0x4, 0x0)
	/usr/local/go/src/bufio/bufio.go:226 +0x24f
io.ReadAtLeast(0x36d1280, 0xc00e465ec0, 0xc0054b38b8, 0x9, 0x9, 0x9, 0xc0001a8535, 0x57b9d2d763d01, 0xc300000000000000)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
net/http.http2readFrameHeader(0xc0054b38b8, 0x9, 0x9, 0x36d1280, 0xc00e465ec0, 0x0, 0xc000000000, 0x48c1dc, 0xc00522ac10)
	/usr/local/go/src/net/http/h2_bundle.go:1479 +0x87
net/http.(*http2Framer).ReadFrame(0xc0054b3880, 0xc00046f560, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:1737 +0xa1
net/http.(*http2clientConnReadLoop).run(0xc00532efa8, 0x4, 0xc00da2cb60)
	/usr/local/go/src/net/http/h2_bundle.go:8246 +0x8d
net/http.(*http2ClientConn).readLoop(0xc00d18b500)
	/usr/local/go/src/net/http/h2_bundle.go:8174 +0x6f
created by net/http.(*http2Transport).newClientConn
	/usr/local/go/src/net/http/h2_bundle.go:7174 +0x64a

goroutine 430 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00060bf90, 0x1, 0x0, 0x0, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:395 +0x122
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0005f3aa0, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:513 +0x1cd
google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc000023200)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:296 +0xcb
created by google.golang.org/grpc/internal/transport.newHTTP2Server
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:293 +0x10c1

goroutine 431 [select]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc000023200)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:968 +0x285
created by google.golang.org/grpc/internal/transport.newHTTP2Server
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:302 +0x10e6

goroutine 432 [IO wait]:
internal/poll.runtime_pollWait(0x7f39ff554c20, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc0056a0618, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0056a0600, 0xc0006e6000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc0056a0600, 0xc0006e6000, 0x8000, 0x8000, 0x203004, 0x0, 0x60100000008)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0000106d0, 0xc0006e6000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
bufio.(*Reader).Read(0xc0007ab200, 0xc0000b7458, 0x9, 0x9, 0x4779a60, 0x1007f39ff6cce98, 0x0)
	/usr/local/go/src/bufio/bufio.go:226 +0x24f
io.ReadAtLeast(0x36d1280, 0xc0007ab200, 0xc0000b7458, 0x9, 0x9, 0x9, 0x5ec0fa52, 0xc00532ddc8, 0x4d7a76)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc0000b7458, 0x9, 0x9, 0x36d1280, 0xc0007ab200, 0x0, 0x0, 0x0, 0x0)
	/work/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0000b7420, 0xc0090fa8e0, 0x47782a0, 0x0, 0x0)
	/work/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc000023200, 0xc00068b500, 0x3392dc0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:458 +0x98
google.golang.org/grpc.(*Server).serveStreams(0xc00020e000, 0x372cce0, 0xc000023200)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:718 +0xdb
google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00020e000, 0x372cce0, 0xc000023200)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:679 +0x3f
created by google.golang.org/grpc.(*Server).handleRawConn
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:678 +0x562

goroutine 466 [select, 8292 minutes]:
github.com/dfuse-io/manageos/superviser.(*Superviser).start(0xc00066a120, 0xc000598580)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/superviser/superviser.go:163 +0x16f
created by github.com/dfuse-io/manageos/superviser.(*Superviser).Start
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/superviser/superviser.go:111 +0x48e

goroutine 470 [select]:
github.com/dfuse-io/bstream/blockstream.(*Server).Blocks(0xc0006145a0, 0xc00060a1e0, 0x3726200, 0xc0057cbe40, 0x0, 0x0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/blockstream/server.go:95 +0x420
github.com/dfuse-io/pbgo/dfuse/bstream/v1._BlockStream_Blocks_Handler(0x1c87ec0, 0xc0006145a0, 0x37232c0, 0xc003e962e0, 0xc00068abd0, 0x6)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/dfuse/bstream/v1/bstream.pb.go:462 +0x109
github.com/dfuse-io/dgrpc.setupLoggingInterceptors.func2(0x1c87ec0, 0xc0006145a0, 0x37232c0, 0xc003e962e0, 0xc003e961c0, 0x33904e0, 0x0, 0x0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/logging.go:68 +0xd1
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x1c87ec0, 0xc0006145a0, 0x37232c0, 0xc003e962e0, 0x3716140, 0xc00068aae0)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x5f
github.com/dfuse-io/dgrpc.setupTracingInterceptors.func2(0x1c87ec0, 0xc0006145a0, 0x37232c0, 0xc003e962e0, 0xc003e961c0, 0xc003e961e0, 0x3716101, 0xc003e962e0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/tracing.go:39 +0xe8
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x1c87ec0, 0xc0006145a0, 0x37232c0, 0xc003e962e0, 0x24, 0xbfa67689ea54a395)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x5f
github.com/grpc-ecosystem/go-grpc-middleware/logging/zap.StreamServerInterceptor.func1(0x1c87ec0, 0xc0006145a0, 0x3723440, 0xc003e96280, 0xc003e961c0, 0xc003e96200, 0xc000318901, 0xc003e96280)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/logging/zap/server_interceptors.go:59 +0x16a
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x1c87ec0, 0xc0006145a0, 0x3723440, 0xc003e96280, 0x24, 0xc00060a190)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x5f
github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).StreamServerInterceptor.func1(0x1c87ec0, 0xc0006145a0, 0x3723320, 0xc000148340, 0xc003e961c0, 0xc003e96220, 0xc00019a001, 0xc000148340)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/server_metrics.go:121 +0xeb
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x1c87ec0, 0xc0006145a0, 0x3723320, 0xc000148340, 0x7f39ff6cd560, 0x0)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x5f
github.com/grpc-ecosystem/go-grpc-middleware/tags.StreamServerInterceptor.func1(0x1c87ec0, 0xc0006145a0, 0x3723860, 0xc0000120c0, 0xc003e961c0, 0xc003e96240, 0xc003e96260, 0xc003e961c0)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/tags/interceptors.go:38 +0x265
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x1c87ec0, 0xc0006145a0, 0x3723860, 0xc0000120c0, 0xc00065bc50, 0x418738)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x5f
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1(0x1c87ec0, 0xc0006145a0, 0x3723860, 0xc0000120c0, 0xc003e961c0, 0x33904e0, 0x3716140, 0xc00068a900)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:58 +0xcf
google.golang.org/grpc.(*Server).processStreamingRPC(0xc00020e000, 0x372cce0, 0xc0006d5200, 0xc000662000, 0xc000552b10, 0x46a2440, 0x0, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:1244 +0x505
google.golang.org/grpc.(*Server).handleStream(0xc00020e000, 0x372cce0, 0xc0006d5200, 0xc000662000, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:1317 +0xcd6
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0003b4080, 0xc00020e000, 0x372cce0, 0xc0006d5200, 0xc000662000)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:722 +0xa1
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:720 +0xa1

goroutine 5512207167 [chan receive]:
github.com/dfuse-io/dfuse-eosio/codec.(*decodingQueue).drainQueueFully(0xc0114c4740, 0x738464a)
	/work/src/codec/abi_decoder.go:450 +0x1a8
created by github.com/dfuse-io/dfuse-eosio/codec.newDecodingQueue
	/work/src/codec/abi_decoder.go:378 +0x153

goroutine 475 [select]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc003ef8600)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:968 +0x285
created by google.golang.org/grpc/internal/transport.newHTTP2Server
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:302 +0x10e6

goroutine 476 [IO wait]:
internal/poll.runtime_pollWait(0x7f39d51ed2a0, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000660898, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000660880, 0xc0007a2000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc000660880, 0xc0007a2000, 0x8000, 0x8000, 0x800010601, 0x0, 0x1060100000000)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00019a258, 0xc0007a2000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
bufio.(*Reader).Read(0xc000532ae0, 0xc000638038, 0x9, 0x9, 0xc00d294a80, 0x7f39ff6cc7d0, 0x0)
	/usr/local/go/src/bufio/bufio.go:226 +0x24f
io.ReadAtLeast(0x36d1280, 0xc000532ae0, 0xc000638038, 0x9, 0x9, 0x9, 0x9436d5, 0xc0049a938c, 0xc01192e040)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc000638038, 0x9, 0x9, 0x36d1280, 0xc000532ae0, 0x0, 0x0, 0xc0049a9380, 0x0)
	/work/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000638000, 0xc0049a9380, 0x47782a0, 0x0, 0x0)
	/work/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc003ef8600, 0xc00068b0e0, 0x3392dc0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:458 +0x98
google.golang.org/grpc.(*Server).serveStreams(0xc00020e000, 0x372cce0, 0xc003ef8600)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:718 +0xdb
google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00020e000, 0x372cce0, 0xc003ef8600)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:679 +0x3f
created by google.golang.org/grpc.(*Server).handleRawConn
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:678 +0x562

goroutine 588 [IO wait]:
internal/poll.runtime_pollWait(0x7f39d51ed1c0, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000201e18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000201e00, 0xc00065c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc000201e00, 0xc00065c000, 0x1000, 0x1000, 0x4, 0xc003b98980, 0x4)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0006685f0, 0xc00065c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
net/http.(*connReader).Read(0xc003dd2150, 0xc00065c000, 0x1000, 0x1000, 0x7f39d5bc2450, 0x0, 0x203001)
	/usr/local/go/src/net/http/server.go:786 +0xf4
bufio.(*Reader).fill(0xc00074c000)
	/usr/local/go/src/bufio/bufio.go:100 +0x103
bufio.(*Reader).ReadSlice(0xc00074c000, 0xa, 0x7f39d5bc2450, 0xc00068f9c0, 0x417ec6, 0xc00d194400, 0x100)
	/usr/local/go/src/bufio/bufio.go:359 +0x3d
bufio.(*Reader).ReadLine(0xc00074c000, 0xc00068f9c8, 0xc003dc4a80, 0x7f39ff6cd560, 0x0, 0x418738, 0x30)
	/usr/local/go/src/bufio/bufio.go:388 +0x34
net/textproto.(*Reader).readLineSlice(0xc004b5a450, 0xc00d194400, 0x0, 0x442d9c, 0xc00068fa28, 0x0)
	/usr/local/go/src/net/textproto/reader.go:58 +0x6c
net/textproto.(*Reader).ReadLine(...)
	/usr/local/go/src/net/textproto/reader.go:39
net/http.readRequest(0xc00074c000, 0x0, 0xc00d194400, 0x0, 0x0)
	/usr/local/go/src/net/http/request.go:1015 +0xa4
net/http.(*conn).readRequest(0xc000081360, 0x3716080, 0xc000555200, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:966 +0x191
net/http.(*conn).serve(0xc000081360, 0x3716080, 0xc000555200)
	/usr/local/go/src/net/http/server.go:1822 +0x6d4
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2933 +0x35c

goroutine 5512212130 [chan receive]:
cloud.google.com/go/storage.(*Writer).Close(0xc00ce54900, 0x0, 0x0)
	/work/go/pkg/mod/cloud.google.com/go/[email protected]/writer.go:222 +0x92
github.com/dfuse-io/dstore.(*GSStore).WriteObject(0xc000111780, 0x3716100, 0xc00f3354a0, 0xc00ea41460, 0x2e, 0x36d66e0, 0xc00f6a4160, 0x518872, 0xc00f3354b0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/gsstore.go:68 +0x373
github.com/dfuse-io/dstore.pushLocalFile(0x3716100, 0xc00f3354a0, 0x372f8e0, 0xc000111780, 0xc00ea41440, 0x52, 0xc00ea41460, 0x2e, 0x0, 0x0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/common.go:43 +0x23a
github.com/dfuse-io/dstore.(*GSStore).PushLocalFile(0xc000111780, 0x3716100, 0xc00f3354a0, 0xc00ea41440, 0x52, 0xc00ea41460, 0x2e, 0x1, 0xc0000b9518)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/gsstore.go:117 +0x80
github.com/dfuse-io/manageos/mindreader.(*DefaultArchiver).uploadFiles.func1(0x0, 0x0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/mindreader/archiver.go:142 +0x259
github.com/abourget/llerrgroup.(*Group).Go.func1(0x100000000000000, 0xc00cde97a0)
	/work/go/pkg/mod/github.com/abourget/[email protected]/group.go:74 +0x33
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc00029ea80, 0xc00d1feb40)
	/work/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57 +0x59
created by golang.org/x/sync/errgroup.(*Group).Go
	/work/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:54 +0x66

goroutine 9502 [IO wait]:
internal/poll.runtime_pollWait(0x7f39ff554a60, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000696698, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000696680, 0xc0005c8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc000696680, 0xc0005c8000, 0x1000, 0x1000, 0x4, 0xc00a61f4c0, 0x4)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00019a290, 0xc0005c8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
net/http.(*connReader).Read(0xc00f57af60, 0xc0005c8000, 0x1000, 0x1000, 0x7f39b4d99878, 0x0, 0x203002)
	/usr/local/go/src/net/http/server.go:786 +0xf4
bufio.(*Reader).fill(0xc000532e40)
	/usr/local/go/src/bufio/bufio.go:100 +0x103
bufio.(*Reader).ReadSlice(0xc000532e40, 0xa, 0x7f39b4d99878, 0xc0002c19c0, 0x417ec6, 0xc00c922a00, 0x100)
	/usr/local/go/src/bufio/bufio.go:359 +0x3d
bufio.(*Reader).ReadLine(0xc000532e40, 0xc0002c19c8, 0xc005256a80, 0x7f39ff6cce98, 0x0, 0x418738, 0x30)
	/usr/local/go/src/bufio/bufio.go:388 +0x34
net/textproto.(*Reader).readLineSlice(0xc00a2f2090, 0xc00c922a00, 0x0, 0x442d9c, 0xc0002c1a28, 0x0)
	/usr/local/go/src/net/textproto/reader.go:58 +0x6c
net/textproto.(*Reader).ReadLine(...)
	/usr/local/go/src/net/textproto/reader.go:39
net/http.readRequest(0xc000532e40, 0x0, 0xc00c922a00, 0x0, 0x0)
	/usr/local/go/src/net/http/request.go:1015 +0xa4
net/http.(*conn).readRequest(0xc0007eaaa0, 0x3716080, 0xc011d76400, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:966 +0x191
net/http.(*conn).serve(0xc0007eaaa0, 0x3716080, 0xc011d76400)
	/usr/local/go/src/net/http/server.go:1822 +0x6d4
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2933 +0x35c

goroutine 5491356689 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00fff1360, 0x1, 0x0, 0x0, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:395 +0x122
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00ffe2fc0, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:513 +0x1cd
google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc00acb2fc0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:346 +0x7b
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:344 +0xedb

goroutine 5511303334 [select, 1 minutes]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00f6de3c0, 0x1, 0x0, 0x0, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:395 +0x122
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00c731c20, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:513 +0x1cd
google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc00078d6c0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:346 +0x7b
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:344 +0xedb

goroutine 20884 [IO wait]:
internal/poll.runtime_pollWait(0x7f39ff555240, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc0053943d8, 0x72, 0x8001, 0x8000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0053943c0, 0xc005988000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
os.(*File).read(...)
	/usr/local/go/src/os/file_unix.go:263
os.(*File).Read(0xc011ad5790, 0xc005988000, 0x8000, 0x8000, 0xf3, 0x0, 0x0)
	/usr/local/go/src/os/file.go:116 +0x71
io.copyBuffer(0x36d2140, 0xc0056ec3c0, 0x36d66e0, 0xc011ad5790, 0xc005988000, 0x8000, 0x8000, 0x370e901, 0xc000482fc8, 0x778293)
	/usr/local/go/src/io/io.go:405 +0x122
io.Copy(...)
	/usr/local/go/src/io/io.go:364
os/exec.(*Cmd).writerDescriptor.func1(0x3393a20, 0xc005396140)
	/usr/local/go/src/os/exec/exec.go:311 +0x63
os/exec.(*Cmd).Start.func1(0xc005367340, 0xc00d968c20)
	/usr/local/go/src/os/exec/exec.go:441 +0x27
created by os/exec.(*Cmd).Start
	/usr/local/go/src/os/exec/exec.go:440 +0x632

goroutine 5512218863 [runnable]:
github.com/eoscanada/eos-go.(*Decoder).ReadName(0xc0002bf260, 0xc016a41ec0, 0xc, 0x0, 0x0)
	/work/go/pkg/mod/github.com/eoscanada/[email protected]/decoder.go:1099 +0x21c
github.com/dfuse-io/dfuse-eosio/codec.decodeTransfer(0xc016cf9108, 0x21, 0x50ef8, 0x8, 0x0, 0xc00a333ea0, 0x0)
	/work/src/codec/abi_decoder.go:573 +0x17e
github.com/dfuse-io/dfuse-eosio/codec.(*decodingQueue).decodeAction(0xc0114c4740, 0xc016e2a800, 0x1ce94e43e2, 0xc0169ca4c0, 0x40, 0x738464a, 0x0, 0xc00d059980, 0x7f39d5bbd8a0)
	/work/src/codec/abi_decoder.go:482 +0x3b7
github.com/dfuse-io/dfuse-eosio/codec.(*decodingQueue).executeDecodingJob(0xc0114c4740, 0x37160c0, 0xc000198020, 0xc016a56150, 0x1, 0x1, 0xc00b94c7a0, 0x91ab24, 0xc011274000, 0xc004f2a000, ...)
	/work/src/codec/abi_decoder.go:434 +0x523
github.com/dfuse-io/dhammer.(*Hammer).processBatch(0xc009aaa500, 0x37160c0, 0xc000198020, 0xc016a56150, 0x1, 0x1, 0xc016a4d0e0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/dhammer.go:164 +0xae
created by github.com/dfuse-io/dhammer.(*Hammer).runInput
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/dhammer.go:152 +0x41a

goroutine 5512207165 [select]:
github.com/dfuse-io/dhammer.(*Hammer).runInput(0xc009aaa500, 0x37160c0, 0xc000198020)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/dhammer.go:143 +0x38b
created by github.com/dfuse-io/dhammer.(*Hammer).Start
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/dhammer.go:93 +0x57

goroutine 5491356688 [IO wait]:
internal/poll.runtime_pollWait(0x7f39ff5555c0, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc00d8e1498, 0x72, 0x1000, 0x1019, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00d8e1480, 0xc00d806600, 0x1019, 0x1019, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc00d8e1480, 0xc00d806600, 0x1019, 0x1019, 0x203001, 0x6b7480, 0xc00c5a2138)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00d636298, 0xc00d806600, 0x1019, 0x1019, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
crypto/tls.(*atLeastReader).Read(0xc004614cc0, 0xc00d806600, 0x1019, 0x1019, 0x22, 0x1014, 0xc00532ca48)
	/usr/local/go/src/crypto/tls/conn.go:760 +0x60
bytes.(*Buffer).ReadFrom(0xc00c5a2258, 0x36d1540, 0xc004614cc0, 0x415e15, 0x1b3b2a0, 0x1d9fb40)
	/usr/local/go/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc00c5a2000, 0x36d62a0, 0xc00d636298, 0x5, 0xc00d636298, 0x11)
	/usr/local/go/src/crypto/tls/conn.go:782 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc00c5a2000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:589 +0x115
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:557
crypto/tls.(*Conn).Read(0xc00c5a2000, 0xc010274000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1233 +0x15b
bufio.(*Reader).Read(0xc00ffe2f00, 0xc004ae2658, 0x9, 0x9, 0x4779a60, 0x7f39ff6cd560, 0x0)
	/usr/local/go/src/bufio/bufio.go:226 +0x24f
io.ReadAtLeast(0x36d1280, 0xc00ffe2f00, 0xc004ae2658, 0x9, 0x9, 0x9, 0x462713, 0x744a70ef851c0, 0xc00532ce50)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc004ae2658, 0x9, 0x9, 0x36d1280, 0xc00ffe2f00, 0x0, 0xbfa85c7300000000, 0x1c48b51db4d90, 0x47782a0)
	/work/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc004ae2620, 0xc00d40c600, 0xc00d40c600, 0x0, 0x0)
	/work/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00acb2fc0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1264 +0x16f
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:300 +0xd23

goroutine 5512212132 [select]:
net/http.(*http2ClientConn).roundTrip(0xc00d18b500, 0xc015013200, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:7596 +0x977
net/http.(*http2Transport).RoundTripOpt(0xc00019cf60, 0xc015013200, 0x1af0c00, 0xc000798930, 0xc00d31b380, 0x5)
	/usr/local/go/src/net/http/h2_bundle.go:6948 +0x16f
net/http.(*http2Transport).RoundTrip(...)
	/usr/local/go/src/net/http/h2_bundle.go:6909
net/http.http2noDialH2RoundTripper.RoundTrip(0xc00019cf60, 0xc015013200, 0x36d8460, 0xc00019cf60, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:9103 +0x3e
net/http.(*Transport).roundTrip(0x46af0e0, 0xc015013200, 0x1a5a980, 0xc00a76fe01, 0xc003ee0420)
	/usr/local/go/src/net/http/transport.go:515 +0xd94
net/http.(*Transport).RoundTrip(0x46af0e0, 0xc015013200, 0x1e2335d, 0xa, 0xc003ee0508)
	/usr/local/go/src/net/http/roundtrip.go:17 +0x35
google.golang.org/api/transport/http.parameterTransport.RoundTrip(0xc00011cae0, 0x1e, 0x0, 0x0, 0x0, 0x0, 0x36d6340, 0x46af0e0, 0xc015013100, 0x1e3dbd9, ...)
	/work/go/pkg/mod/google.golang.org/[email protected]/transport/http/dial.go:130 +0x235
go.opencensus.io/plugin/ochttp.(*traceTransport).RoundTrip(0xc00fb56880, 0xc015013100, 0xc005019c80, 0x1, 0x1)
	/work/go/pkg/mod/[email protected]/plugin/ochttp/trace.go:84 +0x456
go.opencensus.io/plugin/ochttp.statsTransport.RoundTrip(0x36d3f60, 0xc00fb56880, 0xc015012e00, 0xc00fb5a440, 0xc00fb5a440, 0x1be9ac0)
	/work/go/pkg/mod/[email protected]/plugin/ochttp/client_stats.go:57 +0x5cc
go.opencensus.io/plugin/ochttp.(*Transport).RoundTrip(0xc000614550, 0xc015012e00, 0x0, 0x0, 0xc005256a80)
	/work/go/pkg/mod/[email protected]/plugin/ochttp/client.go:99 +0x1fb
golang.org/x/oauth2.(*Transport).RoundTrip(0xc0001116e0, 0xc015012d00, 0x0, 0x0, 0x0)
	/work/go/pkg/mod/golang.org/x/[email protected]/transport.go:55 +0x155
net/http.send(0xc015012d00, 0x36d40c0, 0xc0001116e0, 0x0, 0x0, 0x0, 0xc00f6a4190, 0xc00d31b3bb, 0x1, 0x0)
	/usr/local/go/src/net/http/client.go:252 +0x43e
net/http.(*Client).send(0xc0005529c0, 0xc015012d00, 0x0, 0x0, 0x0, 0xc00f6a4190, 0x0, 0x1, 0xc005256a80)
	/usr/local/go/src/net/http/client.go:176 +0xfa
net/http.(*Client).do(0xc0005529c0, 0xc015012d00, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/client.go:699 +0x44a
net/http.(*Client).Do(...)
	/usr/local/go/src/net/http/client.go:567
google.golang.org/api/internal/gensupport.send(0x3716100, 0xc00f3354a0, 0xc0005529c0, 0xc015012c00, 0x47a6b80, 0xc00070fa00, 0x91fd46)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/gensupport/send.go:67 +0xef
google.golang.org/api/internal/gensupport.SendRequest(0x3716100, 0xc00f3354a0, 0xc0005529c0, 0xc015012c00, 0xc00070fd08, 0x7d, 0x7f39d51ebe40)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/gensupport/send.go:52 +0x18d
google.golang.org/api/storage/v1.(*ObjectsInsertCall).doRequest(0xc00e640410, 0x1e1b11d, 0x4, 0x0, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/storage/v1/storage-gen.go:10041 +0x8cc
google.golang.org/api/storage/v1.(*ObjectsInsertCall).Do(0xc00e640410, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/storage/v1/storage-gen.go:10053 +0xbd
cloud.google.com/go/storage.(*Writer).open.func1(0xc00ce54900, 0xc00f794000, 0xc00f6a4168, 0xc00d1feba0, 0x2, 0x2)
	/work/go/pkg/mod/cloud.google.com/go/[email protected]/writer.go:160 +0x79b
created by cloud.google.com/go/storage.(*Writer).open
	/work/go/pkg/mod/cloud.google.com/go/[email protected]/writer.go:110 +0x3f3

goroutine 5511303333 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f39d51ecf20, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc00d1fca18, 0x72, 0x1000, 0x1018, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00d1fca00, 0xc00d9b6c00, 0x1018, 0x1018, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc00d1fca00, 0xc00d9b6c00, 0x1018, 0x1018, 0x203001, 0x6b7480, 0xc003dc5638)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00f684a08, 0xc00d9b6c00, 0x1018, 0x1018, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
crypto/tls.(*atLeastReader).Read(0xc004f56e20, 0xc00d9b6c00, 0x1018, 0x1018, 0x1a, 0x1013, 0xc00e06aa48)
	/usr/local/go/src/crypto/tls/conn.go:760 +0x60
bytes.(*Buffer).ReadFrom(0xc003dc5758, 0x36d1540, 0xc004f56e20, 0x415e15, 0x1b3b2a0, 0x1d9fb40)
	/usr/local/go/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc003dc5500, 0x36d62a0, 0xc00f684a08, 0x5, 0xc00f684a08, 0x9)
	/usr/local/go/src/crypto/tls/conn.go:782 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc003dc5500, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:589 +0x115
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:557
crypto/tls.(*Conn).Read(0xc003dc5500, 0xc003bc8000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1233 +0x15b
bufio.(*Reader).Read(0xc00c731b60, 0xc003fb6b98, 0x9, 0x9, 0xc00ab21180, 0x7f39ff6cce98, 0x0)
	/usr/local/go/src/bufio/bufio.go:226 +0x24f
io.ReadAtLeast(0x36d1280, 0xc00c731b60, 0xc003fb6b98, 0x9, 0x9, 0x9, 0x942c05, 0xc0046c8660, 0xc0046c0004)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc003fb6b98, 0x9, 0x9, 0x36d1280, 0xc00c731b60, 0x0, 0x0, 0xc0046c8660, 0x0)
	/work/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc003fb6b60, 0xc0046c8660, 0xc0046c8600, 0x0, 0x0)
	/work/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00078d6c0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1264 +0x16f
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:300 +0xd23

goroutine 20882 [syscall, 8282 minutes]:
syscall.Syscall6(0xf7, 0x1, 0x30, 0xc00586dd08, 0x1000004, 0x0, 0x0, 0x36a0610, 0xaaaaaaaaaaaaaa, 0xc00586dd48)
	/usr/local/go/src/syscall/asm_linux_amd64.s:41 +0x5
os.(*Process).blockUntilWaitable(0xc005910000, 0x410bd2, 0xc00585a298, 0xc00585a2a8)
	/usr/local/go/src/os/wait_waitid.go:31 +0x98
os.(*Process).wait(0xc005910000, 0xc00586de60, 0x410715, 0xc00585a240)
	/usr/local/go/src/os/exec_unix.go:22 +0x39
os.(*Process).Wait(...)
	/usr/local/go/src/os/exec.go:125
os/exec.(*Cmd).Wait(0xc005367340, 0x14, 0x0)
	/usr/local/go/src/os/exec/exec.go:507 +0x60
github.com/ShinyTrinkets/overseer.(*Cmd).run(0xc005398c60)
	/work/go/pkg/mod/github.com/maoueh/[email protected]/cmd.go:449 +0x4d9
created by github.com/ShinyTrinkets/overseer.(*Cmd).Start
	/work/go/pkg/mod/github.com/maoueh/[email protected]/cmd.go:271 +0xce

goroutine 5512207166 [select]:
github.com/dfuse-io/dhammer.(*Hammer).outputSingleBatch(0xc009aaa500, 0x37160c0, 0xc000198020, 0xc016a4d0e0, 0x1, 0x0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/dhammer.go:219 +0x102
github.com/dfuse-io/dhammer.(*Hammer).linearizeOutput(0xc009aaa500, 0x37160c0, 0xc000198020)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/dhammer.go:208 +0x1b0
created by github.com/dfuse-io/dhammer.(*Hammer).Start
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/dhammer.go:94 +0x8d

goroutine 5512212131 [select]:
cloud.google.com/go/storage.(*Writer).monitorCancel(0xc00ce54900)
	/work/go/pkg/mod/cloud.google.com/go/[email protected]/writer.go:232 +0xc7
created by cloud.google.com/go/storage.(*Writer).open
	/work/go/pkg/mod/cloud.google.com/go/[email protected]/writer.go:98 +0x333

And while I don't think it's relevant, here blocks before the crash:

Blocks before crash 
info  2020-05-17T08:48:16.490 nodeos    producer_plugin.cpp:401       on_incoming_block    ] Received block 535e89aa440ff846... #121128518 @ 2020-05-17T08:48:16.500 signed by eosdotwikibp [trxs: 22, lib: 121128184, conf: 0, latency: -9 ms]
info  2020-05-17T08:48:17.075 nodeos    producer_plugin.cpp:401       on_incoming_block    ] Received block 6515612611377c1e... #121128519 @ 2020-05-17T08:48:17.000 signed by eosdotwikibp [trxs: 21, lib: 121128184, conf: 0, latency: 75 ms]
info  2020-05-17T08:48:17.077 nodeos    producer_plugin.cpp:401       on_incoming_block    ] Received block e87cfb414a6baa08... #121128520 @ 2020-05-17T08:48:17.500 signed by eosdotwikibp [trxs: 1, lib: 121128184, conf: 0, latency: -422 ms]
info  2020-05-17T08:48:18.510 nodeos    producer_plugin.cpp:401       on_incoming_block    ] Received block cbc0c3bd54815c14... #121128521 @ 2020-05-17T08:48:18.000 signed by eoseouldotio [trxs: 18, lib: 121128196, conf: 240, latency: 510 ms]

maoueh avatar May 17 '20 11:05 maoueh

Cannot reproduce.

  • Go repository has no report of other issues with this (not after 2018)
  • Issue appeared on reading from a channel (which should be a thread-safe operation)
  • The program was NOT shutting down at that moment (so nothing was trying to close the channel or whatever.. anyway closing the channel would still be OK...)
  • The item read from the channel was a string with its own bytes assigned

@maoueh @abourget do you want to spend more time looking at this ?

sduchesneau avatar May 19 '20 20:05 sduchesneau

No, let's wait for the next occurrence. Let's keep it open for now.

maoueh avatar May 20 '20 19:05 maoueh