hatchet icon indicating copy to clipboard operation
hatchet copied to clipboard

Published 20 hours ago • verified publisher icon hatchet-dev

bug(engine): goroutine leaks

Open steebchen opened this issue 4 months ago • 0 comments

List of goroutine leaks which are currently ignored, but should be investigate if they could become a problem.

Related code:

https://github.com/hatchet-dev/hatchet/blob/00111d823c9b1a2fa40a01e495bc288c4215d0bd/examples/loadtest/cli/cli_e2e_test.go#L54-L57

Goroutine leaks:

  • [ ] go.opencensus.io/stats/view.(*worker).start

    [Goroutine 6 in state select, with go.opencensus.io/stats/view.(*worker).start on top of the stack:
go.opencensus.io/stats/view.(*worker).start(0xc0002feb80)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/[email protected]/stats/view/worker.go:292 +0x128
created by go.opencensus.io/stats/view.init.0 in goroutine 1
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/[email protected]/stats/view/worker.go:34 +0xf4

  • [ ] google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run

     Goroutine 23 in state select, with google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run on top of the stack:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000373500, {0x102fde440, 0xc00004aaf0})
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76 +0x150
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 54
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x1f8

  • [ ] google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run

     Goroutine 24 in state select, with google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run on top of the stack:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000373530, {0x102fde440, 0xc00004ab40})
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76 +0x150
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 54
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x1f8

  • [ ] google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run

     Goroutine 25 in state select, with google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run on top of the stack:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000373560, {0x102fde440, 0xc00004ab90})
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76 +0x150
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 54
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x1f8
 Goroutine 40 in state IO wait, with internal/poll.runtime_pollWait on top of the stack:
internal/poll.runtime_pollWait(0x14d67ce38, 0x72)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0xc00052c0a0, 0xc0000dd200?, 0x0)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/internal/poll/fd_poll_runtime.go:84 +0xb8
internal/poll.(*pollDesc).waitRead(...)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00052c080, {0xc0000dd200, 0x900, 0x900})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/internal/poll/fd_unix.go:164 +0x2e0
net.(*netFD).Read(0xc00052c080, {0xc0000dd200, 0x900, 0x900})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/net/fd_posix.go:55 +0x48
net.(*conn).Read(0xc00051e018, {0xc0000dd200, 0x900, 0x900})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/net/net.go:179 +0x8c
crypto/tls.(*atLeastReader).Read(0xc00069a870, {0xc0000dd200, 0x900, 0x900})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:805 +0x7c
bytes.(*Buffer).ReadFrom(0xc00060a2a8, {0x102fd69b8, 0xc00069a870})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/bytes/buffer.go:211 +0xf4
crypto/tls.(*Conn).readFromUntil(0xc00060a000, {0x14d67cf30?, 0xc00051e018}, 0x5)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:827 +0x18c
crypto/tls.(*Conn).readRecordOrCCS(0xc00060a000, 0x0)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:625 +0x480
crypto/tls.(*Conn).readRecord(...)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:587
crypto/tls.(*Conn).Read(0xc00060a000, {0xc0005e2000, 0x8000, 0x50000c0003bfb38?})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:1369 +0x1c4
bufio.(*Reader).Read(0xc000594d80, {0xc000596200, 0x9, 0x9})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/bufio/bufio.go:244 +0x390
io.ReadAtLeast({0x102fd5e58, 0xc000594d80}, {0xc000596200, 0x9, 0x9}, 0x9)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/io/io.go:335 +0xcc
io.ReadFull(...)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc000596200, 0x9, 0x9}, {0x102fd5e58, 0xc000594d80})
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x68
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0005961c0)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0xbc
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00057c240, 0xc00060a000?)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1587 +0x1e8
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 26
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:398 +0x2234

  • [ ] google.golang.org/grpc/internal/transport.(*controlBuffer).get

     Goroutine 41 in state select, with google.golang.org/grpc/internal/transport.(*controlBuffer).get on top of the stack:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0004e4460, 0x1)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:418 +0x120
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0000e77a0)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:552 +0xf0
google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:452 +0x114
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 26
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:450 +0x288c

  • [ ] google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run

     Goroutine 55 in state select, with google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run on top of the stack:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc00012b920, {0x102fde440, 0xc00004ac30})
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76 +0x150
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 10
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x1f8

  • [ ] google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run

     Goroutine 56 in state select, with google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run on top of the stack:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc00012b950, {0x102fde440, 0xc00004ad70})
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76 +0x150
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 10
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x1f8

  • [ ] google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run

     Goroutine 57 in state select, with google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run on top of the stack:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc00012b980, {0x102fde440, 0xc00004adc0})
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76 +0x150
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 10
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x1f8
 Goroutine 61 in state IO wait, with internal/poll.runtime_pollWait on top of the stack:
internal/poll.runtime_pollWait(0x14d67cd40, 0x72)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0xc00021dda0, 0xc0000ddb00?, 0x0)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/internal/poll/fd_poll_runtime.go:84 +0xb8
internal/poll.(*pollDesc).waitRead(...)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00021dd80, {0xc0000ddb00, 0x900, 0x900})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/internal/poll/fd_unix.go:164 +0x2e0
net.(*netFD).Read(0xc00021dd80, {0xc0000ddb00, 0x900, 0x900})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/net/fd_posix.go:55 +0x48
net.(*conn).Read(0xc00051e0d0, {0xc0000ddb00, 0x900, 0x900})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/net/net.go:179 +0x8c
crypto/tls.(*atLeastReader).Read(0xc000155d10, {0xc0000ddb00, 0x900, 0x900})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:805 +0x7c
bytes.(*Buffer).ReadFrom(0xc00060b0a8, {0x102fd69b8, 0xc000155d10})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/bytes/buffer.go:211 +0xf4
crypto/tls.(*Conn).readFromUntil(0xc00060ae00, {0x14d67cf30?, 0xc00051e0d0}, 0x5)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:827 +0x18c
crypto/tls.(*Conn).readRecordOrCCS(0xc00060ae00, 0x0)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:625 +0x480
crypto/tls.(*Conn).readRecord(...)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:587
crypto/tls.(*Conn).Read(0xc00060ae00, {0xc000440000, 0x8000, 0x50000c0005258b8?})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:1369 +0x1c4
bufio.(*Reader).Read(0xc00018f140, {0xc0000d63c0, 0x9, 0x9})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/bufio/bufio.go:244 +0x390
io.ReadAtLeast({0x102fd5e58, 0xc00018f140}, {0xc0000d63c0, 0x9, 0x9}, 0x9)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/io/io.go:335 +0xcc
io.ReadFull(...)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc0000d63c0, 0x9, 0x9}, {0x102fd5e58, 0xc00018f140})
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x68
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0000d6380)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0xbc
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00057cd80, 0xc00060ae00?)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1587 +0x1e8
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 14
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:398 +0x2234

  • [ ] google.golang.org/grpc/internal/transport.(*controlBuffer).get

     Goroutine 62 in state select, with google.golang.org/grpc/internal/transport.(*controlBuffer).get on top of the stack:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00004b540, 0x1)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:418 +0x120
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000520d20)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:552 +0xf0
google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:452 +0x114
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 14
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:450 +0x288c
]

steebchen avatar Feb 16 '24 14:02 steebchen