[Bug]: Data Race in morpc

[jiangxinmeng1]

Is there an existing issue for the same bug?

• [X] I have checked the existing issues.

Environment

- Version or commit-id (e.g. v0.1.0 or 8b23a93):
- Hardware parameters:
- OS type:
- Others:

Actual Behavior

WARNING: DATA RACE Write at 0x00c000c58d30 by goroutine 350: runtime.racewriterange() :1 +0x29 internal/poll.ignoringEINTRIO() /usr/local/go/src/internal/poll/fd_unix.go:794 +0x44b internal/poll.(*FD).Read() /usr/local/go/src/internal/poll/fd_unix.go:163 +0x26 net.(*netFD).Read() /usr/local/go/src/net/fd_posix.go:55 +0x50 net.(*conn).Read() /usr/local/go/src/net/net.go:183 +0xb0 net.(*TCPConn).Read() :1 +0x64 github.com/fagongzi/goetty/v2/buf.(*ByteBuf).ReadFrom() /home/jiangxinmeng/go/pkg/mod/github.com/fagongzi/goetty/[email protected]/buf/buf.go:457 +0x119 io.copyBuffer() /usr/local/go/src/io/io.go:412 +0x1c2 io.CopyBuffer() /usr/local/go/src/io/io.go:399 +0x92 github.com/fagongzi/goetty/v2.(*baseIO).readFromConn() /home/jiangxinmeng/go/pkg/mod/github.com/fagongzi/goetty/[email protected]/session.go:403 +0x1ab github.com/fagongzi/goetty/v2.(*baseIO).Read() /home/jiangxinmeng/go/pkg/mod/github.com/fagongzi/goetty/[email protected]/session.go:333 +0x210 github.com/matrixorigin/matrixone/pkg/common/morpc.(*remoteBackend).readLoop() /home/jiangxinmeng/go/github.com/matrixone/pkg/common/morpc/backend.go:443 +0x1c5 github.com/matrixorigin/matrixone/pkg/common/morpc.(*remoteBackend).readLoop-fm() :1 +0x4d github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1() /home/jiangxinmeng/go/github.com/matrixone/pkg/common/stopper/stopper.go:252 +0xd1

Previous read at 0x00c000c58d36 by goroutine 418: github.com/matrixorigin/matrixone/pkg/pb/logservice.(*LogRecord).Unmarshal() /home/jiangxinmeng/go/github.com/matrixone/pkg/pb/logservice/logservice.pb.go:7402 +0x41c github.com/matrixorigin/matrixone/pkg/pb/logservice.(*LogRecordResponse).Unmarshal() /home/jiangxinmeng/go/github.com/matrixone/pkg/pb/logservice/logservice.pb.go:8492 +0x965 github.com/matrixorigin/matrixone/pkg/logservice.MustUnmarshal() /home/jiangxinmeng/go/github.com/matrixone/pkg/logservice/utils.go:34 +0x573 github.com/matrixorigin/matrixone/pkg/logservice.(*client).request() /home/jiangxinmeng/go/github.com/matrixone/pkg/logservice/client.go:396 +0x53a github.com/matrixorigin/matrixone/pkg/logservice.(*client).doRead() /home/jiangxinmeng/go/github.com/matrixone/pkg/logservice/client.go:457 +0xb6 github.com/matrixorigin/matrixone/pkg/logservice.(*client).read() /home/jiangxinmeng/go/github.com/matrixone/pkg/logservice/client.go:325 +0xaf github.com/matrixorigin/matrixone/pkg/logservice.(*managedClient).Read() /home/jiangxinmeng/go/github.com/matrixone/pkg/logservice/client.go:166 +0x6e github.com/matrixorigin/matrixone/pkg/vm/engine/tae/logstore/driver/logservicedriver.(*LogServiceDriver).readFromLogService() /home/jiangxinmeng/go/github.com/matrixone/pkg/vm/engine/tae/logstore/driver/logservicedriver/read.go:64 +0x2cd github.com/matrixorigin/matrixone/pkg/vm/engine/tae/logstore/driver/logservicedriver.(*LogServiceDriver).Read() /home/jiangxinmeng/go/github.com/matrixone/pkg/vm/engine/tae/logstore/driver/logservicedriver/read.go:36 +0x84 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/logstore/driver/logservicedriver.TestAppendRead.func2.1() /home/jiangxinmeng/go/github.com/matrixone/pkg/vm/engine/tae/logstore/driver/logservicedriver/driver_test.go:53 +0x107 github.com/panjf2000/ants/v2.(*goWorker).run.func1() /home/jiangxinmeng/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:70 +0xe9

Goroutine 350 (running) created at: github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask() /home/jiangxinmeng/go/github.com/matrixone/pkg/common/stopper/stopper.go:247 +0x164 github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).RunNamedTask() /home/jiangxinmeng/go/github.com/matrixone/pkg/common/stopper/stopper.go:163 +0x171 github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).RunTask() /home/jiangxinmeng/go/github.com/matrixone/pkg/common/stopper/stopper.go:136 +0x199 github.com/matrixorigin/matrixone/pkg/common/morpc.(*remoteBackend).activeReadLoop() /home/jiangxinmeng/go/github.com/matrixone/pkg/common/morpc/backend.go:568 +0x30 github.com/matrixorigin/matrixone/pkg/common/morpc.(*remoteBackend).resetConn() /home/jiangxinmeng/go/github.com/matrixone/pkg/common/morpc/backend.go:548 +0x3e4 github.com/matrixorigin/matrixone/pkg/common/morpc.NewRemoteBackend() /home/jiangxinmeng/go/github.com/matrixone/pkg/common/morpc/backend.go:185 +0x76a github.com/matrixorigin/matrixone/pkg/common/morpc.(*goettyBasedBackendFactory).Create() /home/jiangxinmeng/go/github.com/matrixone/pkg/common/morpc/backend.go:636 +0x91 github.com/matrixorigin/matrixone/pkg/common/morpc.(*client).createBackendLocked() /home/jiangxinmeng/go/github.com/matrixone/pkg/common/morpc/client.go:346 +0x10a github.com/matrixorigin/matrixone/pkg/common/morpc.(*client).maybeInitBackends() /home/jiangxinmeng/go/github.com/matrixone/pkg/common/morpc/client.go:156 +0x244 github.com/matrixorigin/matrixone/pkg/common/morpc.NewClient() /home/jiangxinmeng/go/github.com/matrixone/pkg/common/morpc/client.go:115 +0x2cf github.com/matrixorigin/matrixone/pkg/logservice.getRPCClient() /home/jiangxinmeng/go/github.com/matrixone/pkg/logservice/client.go:485 +0x54d github.com/matrixorigin/matrixone/pkg/logservice.newClient() /home/jiangxinmeng/go/github.com/matrixone/pkg/logservice/client.go:279 +0x4c4 github.com/matrixorigin/matrixone/pkg/logservice.NewClient() /home/jiangxinmeng/go/github.com/matrixone/pkg/logservice/client.go:119 +0x7c github.com/matrixorigin/matrixone/pkg/vm/engine/tae/logstore/driver/logservicedriver.newClient() /home/jiangxinmeng/go/github.com/matrixone/pkg/vm/engine/tae/logstore/driver/logservicedriver/clientpool.go:32 +0x13a github.com/matrixorigin/matrixone/pkg/vm/engine/tae/logstore/driver/logservicedriver.(*clientpool).newClientFactory.func1() /home/jiangxinmeng/go/github.com/matrixone/pkg/vm/engine/tae/logstore/driver/logservicedriver/clientpool.go:79 +0xc9 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/logstore/driver/logservicedriver.newClientPool() /home/jiangxinmeng/go/github.com/matrixone/pkg/vm/engine/tae/logstore/driver/logservicedriver/clientpool.go:71 +0x28c github.com/matrixorigin/matrixone/pkg/vm/engine/tae/logstore/driver/logservicedriver.NewLogServiceDriver() /home/jiangxinmeng/go/github.com/matrixone/pkg/vm/engine/tae/logstore/driver/logservicedriver/driver.go:56 +0x1d6 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/logstore/driver/logservicedriver.TestAppendRead() /home/jiangxinmeng/go/github.com/matrixone/pkg/vm/engine/tae/logstore/driver/logservicedriver/driver_test.go:34 +0x244 testing.tRunner() /usr/local/go/src/testing/testing.go:1439 +0x213 testing.(*T).Run.func1() /usr/local/go/src/testing/testing.go:1486 +0x47

Goroutine 418 (running) created at: github.com/panjf2000/ants/v2.(*goWorker).run() /home/jiangxinmeng/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:48 +0xb6 github.com/panjf2000/ants/v2.(*Pool).retrieveWorker.func1() /home/jiangxinmeng/go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:223 +0xa4 github.com/panjf2000/ants/v2.(*Pool).retrieveWorker() /home/jiangxinmeng/go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:235 +0x161 github.com/panjf2000/ants/v2.(*Pool).Submit() /home/jiangxinmeng/go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:150 +0x52 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/logstore/driver/logservicedriver.TestAppendRead() /home/jiangxinmeng/go/github.com/matrixone/pkg/vm/engine/tae/logstore/driver/logservicedriver/driver_test.go:66 +0x3ed testing.tRunner() /usr/local/go/src/testing/testing.go:1439 +0x213 testing.(*T).Run.func1() /usr/local/go/src/testing/testing.go:1486 +0x47

Expected Behavior

No response

Steps to Reproduce

No response

Additional information

No response

[zhangxu19830126]

This is a bug in client usage, where a client cannot be used to send again until its Future is returned

[zhangxu19830126]

fixed