pd icon indicating copy to clipboard operation
pd copied to clipboard

Published 20 hours ago • verified publisher icon tikv

TestRandomShutdown is unstable

Open HuSharp opened this issue 10 months ago • 1 comments

Flaky Test

TestMicroserviceTSOClient/TestRandomShutdown

Which jobs are failing

2024-04-19T09:02:55.0852245Z [2024/04/19 09:02:55.084 +00:00] [WARN] [grpclog.go:46] ["[core][Channel #163 SubChannel #166] grpc: addrConn.createTransport failed to connect to {Addr: \"127.0.0.1:34669\", ServerName: \"127.0.0.1:34669\", }. Err: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:34669: connect: connection refused\""]
2024-04-19T09:02:56.2446819Z === RUN   TestMicroserviceTSOClient/TestRandomShutdown
2024-04-19T09:02:56.2494552Z     testutil.go:83: 
2024-04-19T09:02:56.2496270Z         	Error Trace:	/home/runner/work/pd/pd/tests/integrations/mcs/testutil.go:83
2024-04-19T09:02:56.2498557Z         	            				/opt/hostedtoolcache/go/1.21.9/x64/src/runtime/asm_amd64.s:1650
2024-04-19T09:02:56.2500053Z         	Error:      	"449188299075223585" is not less than "0"
2024-04-19T09:02:56.2501353Z         	Test:       	TestMicroserviceTSOClient/TestRandomShutdown
2024-04-19T09:02:57.8529652Z [2024/04/19 09:02:57.852 +00:00] [WARN] [grpclog.go:46] ["[core][Channel #163 SubChannel #166] grpc: addrConn.createTransport failed to connect to {Addr: \"127.0.0.1:34669\", ServerName: \"127.0.0.1:34669\", }. Err: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:34669: connect: connection refused\""]
2024-04-19T09:02:59.3457353Z [2024/04/19 09:02:59.345 +00:00] [INFO] [server.go:174] ["closing tso server ..."]

CI link

https://github.com/tikv/pd/actions/runs/8751050232/job/24015860152

Reason for failure (if possible)

Anything else

HuSharp avatar Apr 19 '24 09:04 HuSharp

https://github.com/tikv/pd/actions/runs/10381657086/job/28743439298?pr=8516

=== RUN   TestLegacyTSOClientSuite/TestRandomShutdown
==================
WARNING: DATA RACE
Write at 0x00c00d0921c0 by goroutine 83864:
  github.com/tikv/pd/client.(*tsoClient).getTSORequest()
      /home/runner/work/pd/pd/client/tso_client.go:201 +0x1a4
  github.com/tikv/pd/client.(*client).dispatchTSORequestWithRetry()
      /home/runner/work/pd/pd/client/client.go:892 +0x1d4
  github.com/tikv/pd/client.(*client).GetLocalTSAsync()
      /home/runner/work/pd/pd/client/client.go:865 +0x371
  github.com/tikv/pd/client.(*client).GetTSAsync()
      /home/runner/work/pd/pd/client/client.go:854 +0xd8
  github.com/tikv/pd/client.(*client).GetTS()
      /home/runner/work/pd/pd/client/client.go:910 +0x70
  github.com/tikv/pd/tests/integrations/mcs.CheckMultiKeyspacesTSO.func1()
      /home/runner/work/pd/pd/tests/integrations/mcs/testutil.go:77 +0x18f
  github.com/tikv/pd/tests/integrations/mcs.CheckMultiKeyspacesTSO.func3()
      /home/runner/work/pd/pd/tests/integrations/mcs/testutil.go:86 +0x4f

Previous read at 0x00c00d0921c0 by goroutine 1696:
  github.com/tikv/pd/client.(*tsoBatchController).finishCollectedRequests()
      /home/runner/work/pd/pd/client/tso_batch_controller.go:143 +0x14d
  github.com/tikv/pd/client.(*tsoDispatcher).processRequests()
      /home/runner/work/pd/pd/client/tso_dispatcher.go:443 +0xaa7
  github.com/tikv/pd/client.(*tsoDispatcher).handleDispatcher()
      /home/runner/work/pd/pd/client/tso_dispatcher.go:287 +0xaa5
  github.com/tikv/pd/client.(*tsoClient).createTSODispatcher.func1()
      /home/runner/work/pd/pd/client/tso_client.go:612 +0x44

Goroutine 83864 (running) created at:
  github.com/tikv/pd/tests/integrations/mcs.CheckMultiKeyspacesTSO()
      /home/runner/work/pd/pd/tests/integrations/mcs/testutil.go:66 +0x199
  github.com/tikv/pd/tests/integrations/tso.(*tsoClientTestSuite).TestRandomShutdown()
      /home/runner/work/pd/pd/tests/integrations/tso/client_test.go:425 +0x184
  github.com/pingcap/failpoint.parseTerm()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected][801](https://github.com/tikv/pd/actions/runs/10381657086/job/28743439298?pr=8516#step:4:802)062533-2eaa32854a6c/terms.go:149 +0x364
  github.com/pingcap/failpoint.parse()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:126 +0xa5
  github.com/pingcap/failpoint.newTerms()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:98 +0x3e
  github.com/pingcap/failpoint.(*Failpoint).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoint.go:54 +0x3e
  github.com/pingcap/failpoint.(*Failpoints).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoints.go:105 +0x296
  github.com/pingcap/failpoint.Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoints.go:222 +0x59
  github.com/tikv/pd/tests/integrations/tso.(*tsoClientTestSuite).TestRandomShutdown()
      /home/runner/work/pd/pd/tests/integrations/tso/client_test.go:409 +0x5a
  runtime.call16()
      /opt/hostedtoolcache/go/1.21.12/x64/src/runtime/asm_amd64.s:747 +0x42
  reflect.Value.Call()
      /opt/hostedtoolcache/go/1.21.12/x64/src/reflect/value.go:380 +0xb5
  github.com/stretchr/testify/suite.Run.func1()
      /home/runner/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:197 +0x766
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.12/x64/src/testing/testing.go:1595 +0x261
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.12/x64/src/testing/testing.go:1648 +0x44

Goroutine 1696 (running) created at:
  github.com/tikv/pd/client.(*tsoClient).createTSODispatcher()
      /home/runner/work/pd/pd/client/tso_client.go:612 +0x1dc
  github.com/tikv/pd/client.(*tsoClient).updateTSODispatcher.func1()
      /home/runner/work/pd/pd/client/tso_client.go:632 +0xad
  sync.(*Map).Range()
      /opt/hostedtoolcache/go/1.21.12/x64/src/sync/map.go:476 +0x1db
  github.com/tikv/pd/client.(*tsoClient).updateTSODispatcher()
      /home/runner/work/pd/pd/client/tso_client.go:629 +0x109
  github.com/tikv/pd/client.(*tsoClient).setup()
      /home/runner/work/pd/pd/client/tso_client.go:124 +0x1ea
  github.com/tikv/pd/client.(*client).resetTSOClientLocked()
      /home/runner/work/pd/pd/client/client.go:716 +0x85e
  github.com/tikv/pd/client.(*client).setServiceMode()
      /home/runner/work/pd/pd/client/client.go:678 +0x3e6
  github.com/tikv/pd/client.(*client).setServiceMode-fm()
      <autogenerated>:1 +0x3b
  github.com/tikv/pd/client.(*pdServiceDiscovery).checkServiceModeChanged()
      /home/runner/work/pd/pd/client/pd_service_discovery.go:883 +0x570
  github.com/tikv/pd/client.(*pdServiceDiscovery).Init()
      /home/runner/work/pd/pd/client/pd_service_discovery.go:509 +0x317
  github.com/tikv/pd/client.(*client).setup()
      /home/runner/work/pd/pd/client/client.go:636 +0x14e
  github.com/tikv/pd/client.createClientWithKeyspace()
      /home/runner/work/pd/pd/client/client.go:460 +0x7c4
  github.com/tikv/pd/client.NewClientWithContext()
      /home/runner/work/pd/pd/client/client.go:408 +0xcdb
  github.com/tikv/pd/tests/integrations/tso.(*tsoClientTestSuite).SetupSuite()
      /home/runner/work/pd/pd/tests/integrations/tso/client_test.go:107 +0xb9b
  github.com/stretchr/testify/suite.Run()
      /home/runner/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:154 +0x63e
  github.com/tikv/pd/tests/integrations/tso.TestLegacyTSOClientSuite()
      /home/runner/work/pd/pd/tests/integrations/tso/client_test.go:75 +0xa5
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.12/x64/src/testing/testing.go:1595 +0x261
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.12/x64/src/testing/testing.go:1648 +0x44
==================
==================
WARNING: DATA RACE
Write at 0x00c00d0921e8 by goroutine [838](https://github.com/tikv/pd/actions/runs/10381657086/job/28743439298?pr=8516#step:4:839)64:
  github.com/tikv/pd/client.(*tsoClient).getTSORequest()
      /home/runner/work/pd/pd/client/tso_client.go:203 +0x24e
  github.com/tikv/pd/client.(*client).dispatchTSORequestWithRetry()
      /home/runner/work/pd/pd/client/client.go:892 +0x1d4
  github.com/tikv/pd/client.(*client).GetLocalTSAsync()
      /home/runner/work/pd/pd/client/client.go:865 +0x371
  github.com/tikv/pd/client.(*client).GetTSAsync()
      /home/runner/work/pd/pd/client/client.go:854 +0xd8
  github.com/tikv/pd/client.(*client).GetTS()
      /home/runner/work/pd/pd/client/client.go:910 +0x70
  github.com/tikv/pd/tests/integrations/mcs.CheckMultiKeyspacesTSO.func1()
      /home/runner/work/pd/pd/tests/integrations/mcs/testutil.go:77 +0x18f
  github.com/tikv/pd/tests/integrations/mcs.CheckMultiKeyspacesTSO.func3()
      /home/runner/work/pd/pd/tests/integrations/mcs/testutil.go:86 +0x4f

Previous write at 0x00c00d0921e8 by goroutine 1696:
  github.com/tikv/pd/client.(*tsoBatchController).finishCollectedRequests()
      /home/runner/work/pd/pd/client/tso_batch_controller.go:144 +0x1ee
  github.com/tikv/pd/client.(*tsoDispatcher).processRequests()
      /home/runner/work/pd/pd/client/tso_dispatcher.go:443 +0xaa7
  github.com/tikv/pd/client.(*tsoDispatcher).handleDispatcher()
      /home/runner/work/pd/pd/client/tso_dispatcher.go:287 +0xaa5
  github.com/tikv/pd/client.(*tsoClient).createTSODispatcher.func1()
      /home/runner/work/pd/pd/client/tso_client.go:612 +0x44

Goroutine 83864 (running) created at:
  github.com/tikv/pd/tests/integrations/mcs.CheckMultiKeyspacesTSO()
      /home/runner/work/pd/pd/tests/integrations/mcs/testutil.go:66 +0x199
  github.com/tikv/pd/tests/integrations/tso.(*tsoClientTestSuite).TestRandomShutdown()
      /home/runner/work/pd/pd/tests/integrations/tso/client_test.go:425 +0x184
  github.com/pingcap/failpoint.parseTerm()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected][854](https://github.com/tikv/pd/actions/runs/10381657086/job/28743439298?pr=8516#step:4:855)a6c/terms.go:149 +0x364
  github.com/pingcap/failpoint.parse()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:126 +0xa5
  github.com/pingcap/failpoint.newTerms()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:98 +0x3e
  github.com/pingcap/failpoint.(*Failpoint).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoint.go:54 +0x3e
  github.com/pingcap/failpoint.(*Failpoints).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoints.go:105 +0x296
  github.com/pingcap/failpoint.Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoints.go:222 +0x59
  github.com/tikv/pd/tests/integrations/tso.(*tsoClientTestSuite).TestRandomShutdown()
      /home/runner/work/pd/pd/tests/integrations/tso/client_test.go:409 +0x5a
  runtime.call16()
      /opt/hostedtoolcache/go/1.21.12/x64/src/runtime/asm_amd64.s:747 +0x42
  reflect.Value.Call()
      /opt/hostedtoolcache/go/1.21.12/x64/src/reflect/value.go:380 +0xb5
  github.com/stretchr/testify/suite.Run.func1()
      /home/runner/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:197 +0x766
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.12/x64/src/testing/testing.go:1595 +0x261
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.12/x64/src/testing/testing.go:1648 +0x44

Goroutine 1696 (running) created at:
  github.com/tikv/pd/client.(*tsoClient).createTSODispatcher()
      /home/runner/work/pd/pd/client/tso_client.go:612 +0x1dc
  github.com/tikv/pd/client.(*tsoClient).updateTSODispatcher.func1()
      /home/runner/work/pd/pd/client/tso_client.go:632 +0xad
  sync.(*Map).Range()
      /opt/hostedtoolcache/go/1.21.12/x64/src/sync/map.go:476 +0x1db
  github.com/tikv/pd/client.(*tsoClient).updateTSODispatcher()
      /home/runner/work/pd/pd/client/tso_client.go:629 +0x109
  github.com/tikv/pd/client.(*tsoClient).setup()
      /home/runner/work/pd/pd/client/tso_client.go:124 +0x1ea
  github.com/tikv/pd/client.(*client).resetTSOClientLocked()
      /home/runner/work/pd/pd/client/client.go:716 +0x85e
  github.com/tikv/pd/client.(*client).setServiceMode()
      /home/runner/work/pd/pd/client/client.go:678 +0x3e6
  github.com/tikv/pd/client.(*client).setServiceMode-fm()
      <autogenerated>:1 +0x3b
  github.com/tikv/pd/client.(*pdServiceDiscovery).checkServiceModeChanged()
      /home/runner/work/pd/pd/client/pd_service_discovery.go:883 +0x570
  github.com/tikv/pd/client.(*pdServiceDiscovery).Init()
      /home/runner/work/pd/pd/client/pd_service_discovery.go:509 +0x317
  github.com/tikv/pd/client.(*client).setup()
      /home/runner/work/pd/pd/client/client.go:636 +0x14e
  github.com/tikv/pd/client.createClientWithKeyspace()
      /home/runner/work/pd/pd/client/client.go:460 +0x7c4
  github.com/tikv/pd/client.NewClientWithContext()
      /home/runner/work/pd/pd/client/client.go:408 +0xcdb
  github.com/tikv/pd/tests/integrations/tso.(*tsoClientTestSuite).SetupSuite()
      /home/runner/work/pd/pd/tests/integrations/tso/client_test.go:107 +0xb9b
  github.com/stretchr/testify/suite.Run()
      /home/runner/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:154 +0x63e
  github.com/tikv/pd/tests/integrations/tso.TestLegacyTSOClientSuite()
      /home/runner/work/pd/pd/tests/integrations/tso/client_test.go:75 +0xa5
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.12/x64/src/testing/testing.go:1595 +0x261
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.12/x64/src/testing/testing.go:1648 +0x44
==================
==================
WARNING: DATA RACE
Write at 0x00c00d0921f0 by goroutine 83[864](https://github.com/tikv/pd/actions/runs/10381657086/job/28743439298?pr=8516#step:4:865):
  github.com/tikv/pd/client.(*tsoClient).getTSORequest()
      /home/runner/work/pd/pd/client/tso_client.go:204 +0x267
  github.com/tikv/pd/client.(*client).dispatchTSORequestWithRetry()
      /home/runner/work/pd/pd/client/client.go:892 +0x1d4
  github.com/tikv/pd/client.(*client).GetLocalTSAsync()
      /home/runner/work/pd/pd/client/client.go:[865](https://github.com/tikv/pd/actions/runs/10381657086/job/28743439298?pr=8516#step:4:866) +0x371
  github.com/tikv/pd/client.(*client).GetTSAsync()
      /home/runner/work/pd/pd/client/client.go:854 +0xd8
  github.com/tikv/pd/client.(*client).GetTS()
      /home/runner/work/pd/pd/client/client.go:910 +0x70
  github.com/tikv/pd/tests/integrations/mcs.CheckMultiKeyspacesTSO.func1()
      /home/runner/work/pd/pd/tests/integrations/mcs/testutil.go:77 +0x18f
  github.com/tikv/pd/tests/integrations/mcs.CheckMultiKeyspacesTSO.func3()
      /home/runner/work/pd/pd/tests/integrations/mcs/testutil.go:86 +0x4f

Previous write at 0x00c00d0921f0 by goroutine 1696:
  github.com/tikv/pd/client.(*tsoBatchController).finishCollectedRequests()
      /home/runner/work/pd/pd/client/tso_batch_controller.go:144 +0x209
  github.com/tikv/pd/client.(*tsoDispatcher).processRequests()
      /home/runner/work/pd/pd/client/tso_dispatcher.go:443 +0xaa7
  github.com/tikv/pd/client.(*tsoDispatcher).handleDispatcher()
      /home/runner/work/pd/pd/client/tso_dispatcher.go:287 +0xaa5
  github.com/tikv/pd/client.(*tsoClient).createTSODispatcher.func1()
      /home/runner/work/pd/pd/client/tso_client.go:612 +0x44

Goroutine 83864 (running) created at:
  github.com/tikv/pd/tests/integrations/mcs.CheckMultiKeyspacesTSO()
      /home/runner/work/pd/pd/tests/integrations/mcs/testutil.go:66 +0x199
  github.com/tikv/pd/tests/integrations/tso.(*tsoClientTestSuite).TestRandomShutdown()
      /home/runner/work/pd/pd/tests/integrations/tso/client_test.go:425 +0x184
  github.com/pingcap/failpoint.parseTerm()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:149 +0x364
  github.com/pingcap/failpoint.parse()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:126 +0xa5
  github.com/pingcap/failpoint.newTerms()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:98 +0x3e
  github.com/pingcap/failpoint.(*Failpoint).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoint.go:54 +0x3e
  github.com/pingcap/failpoint.(*Failpoints).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoints.go:105 +0x296
  github.com/pingcap/failpoint.Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoints.go:222 +0x59
  github.com/tikv/pd/tests/integrations/tso.(*tsoClientTestSuite).TestRandomShutdown()
      /home/runner/work/pd/pd/tests/integrations/tso/client_test.go:409 +0x5a
  runtime.call16()
      /opt/hostedtoolcache/go/1.21.12/x64/src/runtime/asm_amd64.s:747 +0x42
  reflect.Value.Call()
      /opt/hostedtoolcache/go/1.21.12/x64/src/reflect/value.go:380 +0xb5
  github.com/stretchr/testify/suite.Run.func1()
      /home/runner/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:197 +0x766
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.12/x64/src/testing/testing.go:1595 +0x261
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.12/x64/src/testing/testing.go:1648 +0x44

Goroutine 1696 (running) created at:
  github.com/tikv/pd/client.(*tsoClient).createTSODispatcher()
      /home/runner/work/pd/pd/client/tso_client.go:612 +0x1dc
  github.com/tikv/pd/client.(*tsoClient).updateTSODispatcher.func1()
      /home/runner/work/pd/pd/client/tso_client.go:632 +0xad
  sync.(*Map).Range()
      /opt/hostedtoolcache/go/1.21.12/x64/src/sync/map.go:476 +0x1db
  github.com/tikv/pd/client.(*tsoClient).updateTSODispatcher()
      /home/runner/work/pd/pd/client/tso_client.go:629 +0x109
  github.com/tikv/pd/client.(*tsoClient).setup()
      /home/runner/work/pd/pd/client/tso_client.go:124 +0x1ea
  github.com/tikv/pd/client.(*client).resetTSOClientLocked()
      /home/runner/work/pd/pd/client/client.go:716 +0x85e
  github.com/tikv/pd/client.(*client).setServiceMode()
      /home/runner/work/pd/pd/client/client.go:678 +0x3e6
  github.com/tikv/pd/client.(*client).setServiceMode-fm()
      <autogenerated>:1 +0x3b
  github.com/tikv/pd/client.(*pdServiceDiscovery).checkServiceModeChanged()
      /home/runner/work/pd/pd/client/pd_service_discovery.go:[883](https://github.com/tikv/pd/actions/runs/10381657086/job/28743439298?pr=8516#step:4:884) +0x570
  github.com/tikv/pd/client.(*pdServiceDiscovery).Init()
      /home/runner/work/pd/pd/client/pd_service_discovery.go:509 +0x317
  github.com/tikv/pd/client.(*client).setup()
      /home/runner/work/pd/pd/client/client.go:636 +0x14e
  github.com/tikv/pd/client.createClientWithKeyspace()
      /home/runner/work/pd/pd/client/client.go:460 +0x7c4
  github.com/tikv/pd/client.NewClientWithContext()
      /home/runner/work/pd/pd/client/client.go:408 +0xcdb
  github.com/tikv/pd/tests/integrations/tso.(*tsoClientTestSuite).SetupSuite()
      /home/runner/work/pd/pd/tests/integrations/tso/client_test.go:107 +0xb9b
  github.com/stretchr/testify/suite.Run()
      /home/runner/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:154 +0x63e
  github.com/tikv/pd/tests/integrations/tso.TestLegacyTSOClientSuite()
      /home/runner/work/pd/pd/tests/integrations/tso/client_test.go:75 +0xa5
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.12/x64/src/testing/testing.go:1595 +0x261
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.12/x64/src/testing/testing.go:1648 +0x44
==================
    testutil.go:83: 
        	Error Trace:	/home/runner/work/pd/pd/tests/integrations/mcs/testutil.go:83
        	            				/opt/hostedtoolcache/go/1.21.12/x64/src/runtime/asm_amd64.s:1650
        	Error:      	"0" is not less than "0"
        	Test:       	TestLegacyTSOClientSuite/TestRandomShutdown

okJiang avatar Aug 14 '24 06:08 okJiang