Indexer Crashed With POTENTIAL DEADLOCK on Catchup

[phil-vuollet-anterdit]

Subject of the issue

Your environment

indexer 2.13.0 gp3 16000 IOPS running in docker ubuntu:20.04 https://github.com/algorand/indexer/releases/download/2.13.0/algorand-indexer_linux_amd64_2.13.0.tar.bz2 Main-Net {"level":"info","msg":"Starting catchup service with catchpoint: 22580000#DZIIDQRE67B5YRQYSLF6W7AI4XUUOUDBOUQWJCGXJCQ4ARQN4VRQ","time":"2022-08-03T16:07:03Z"}

Steps to reproduce

Expected behaviour

indexer catches up without fail.

Actual behaviour

fails with the following log messages

Aug 03 10:15:35    {"level":"info","msg":"catchup phase 2 of 4 (Verified Accounts):  5229645 / 14263996","time":"2022-08-03T17:13:10Z"}
Aug 03 10:15:35   POTENTIAL DEADLOCK:
Aug 03 10:15:35   Previous place where the lock was grabbed
Aug 03 10:15:35   goroutine 3135619 lock 0xc0001d6380
Aug 03 10:15:35   go/src/github.com/algorand/indexer/third_party/go-algorand/network/connPerfMon.go:177 network.(*connectionPerformanceMonitor).Notify ??? <<<<<
Aug 03 10:15:35   go/src/github.com/algorand/indexer/third_party/go-algorand/network/wsPeer.go:441 network.(*wsPeer).readLoop ???
Aug 03 10:15:35   Have been trying to lock it again for more than 30s
Aug 03 10:15:35   goroutine 2917933 lock 0xc0001d6380
Aug 03 10:15:35   go/src/github.com/algorand/indexer/third_party/go-algorand/network/connPerfMon.go:177 network.(*connectionPerformanceMonitor).Notify ??? <<<<<
Aug 03 10:15:35   go/src/github.com/algorand/indexer/third_party/go-algorand/network/wsPeer.go:441 network.(*wsPeer).readLoop ???
Aug 03 10:15:35   Here is what goroutine 3135619 doing now
Aug 03 10:15:35   goroutine 3135619 [semacquire]:
Aug 03 10:15:35   sync.runtime_SemacquireMutex(0x23794a0, 0x8c, 0x23794b0)
Aug 03 10:15:35   	/usr/local/go/src/runtime/sema.go:71 +0x25
Aug 03 10:15:35   sync.(*Mutex).lockSlow(0xc000190078)
Aug 03 10:15:35   	/usr/local/go/src/sync/mutex.go:138 +0x165
Aug 03 10:15:35   sync.(*Mutex).Lock(...)
Aug 03 10:15:35   	/usr/local/go/src/sync/mutex.go:81
Aug 03 10:15:35   github.com/algorand/go-deadlock.(*lockOrder).postUnlock(0xc000190078, {0x17f5140, 0xc0001d6380})
Aug 03 10:15:35   	/go/pkg/mod/github.com/algorand/[email protected]/deadlock.go:426 +0x46
Aug 03 10:15:35   github.com/algorand/go-deadlock.postUnlock(...)
Aug 03 10:15:35   	/go/pkg/mod/github.com/algorand/[email protected]/deadlock.go:181
Aug 03 10:15:35   github.com/algorand/go-deadlock.(*Mutex).Unlock(0xc0001d6380)
Aug 03 10:15:35   	/go/pkg/mod/github.com/algorand/[email protected]/deadlock.go:92 +0x5e
Aug 03 10:15:35   github.com/algorand/go-algorand/network.(*connectionPerformanceMonitor).Notify(0xc0001d6380, 0xc000a74ee0)
Aug 03 10:15:35   	/go/src/github.com/algorand/indexer/third_party/go-algorand/network/connPerfMon.go:196 +0x118
Aug 03 10:15:35   github.com/algorand/go-algorand/network.(*wsPeer).readLoop(0xc00c938000)
Aug 03 10:15:35   	/go/src/github.com/algorand/indexer/third_party/go-algorand/network/wsPeer.go:441 +0x4a5
Aug 03 10:15:35   created by github.com/algorand/go-algorand/network.(*wsPeer).init
Aug 03 10:15:35   	/go/src/github.com/algorand/indexer/third_party/go-algorand/network/wsPeer.go:360 +0x3b5