cockroach icon indicating copy to clipboard operation
cockroach copied to clipboard
verified publisher icon cockroachdb

sql: race in TestCancelMultipleQueued under stress

Open yuzefovich opened this issue 3 years ago • 3 comments

I hit a race on one of my branches, but I don't think that my change (#81995) is to blame. This was on c1d005cd7ddf6a52305a211f79a1b80f9317cca8.

==================
WARNING: DATA RACE
Write at 0x00c001287ee8 by goroutine 2068:
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*NumVal).AsInt64()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/constant.go:217 +0x58
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*NumVal).AvailableTypes()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/constant.go:281 +0x30
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.canConstantBecome()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/constant.go:105 +0x41
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.typeCheckOverloadedExprs.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/overload.go:651 +0x94
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.filterOverloads()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/overload.go:1088 +0xe4
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.typeCheckOverloadedExprs()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/overload.go:649 +0x7e9
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*BinaryExpr).TypeCheck()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/type_check.go:316 +0x235
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*ParenExpr).TypeCheck()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/type_check.go:1407 +0x6f
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*CastExpr).TypeCheck()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/type_check.go:578 +0x3a2
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.typeCheckOverloadedExprs()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/overload.go:679 +0xa59
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*BinaryExpr).TypeCheck()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/type_check.go:316 +0x235
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.typeCheckOverloadedExprs()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/overload.go:679 +0xa59
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.typeCheckComparisonOp()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/type_check.go:2237 +0x177e
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*ComparisonExpr).TypeCheck()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/type_check.go:862 +0x217
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.typeCheckAndRequire()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/type_check.go:1956 +0xa4
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.typeCheckAndRequireBoolean()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/type_check.go:1950 +0xcf
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*IfExpr).TypeCheck()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/type_check.go:1285 +0x5b
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.TypeCheckSameTypedExprs()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/type_check.go:2375 +0x6cd
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.typeCheckOverloadedExprs()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/overload.go:607 +0x2b19
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*FuncExpr).TypeCheck()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/type_check.go:1080 +0x6e4
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*CastExpr).TypeCheck()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/type_check.go:578 +0x3a2
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.typeCheckOverloadedExprs()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/overload.go:679 +0xa59
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*BinaryExpr).TypeCheck()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/type_check.go:316 +0x235
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.TypeCheck()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/type_check.go:272 +0x12e
  github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*scope).resolveType()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/scope.go:463 +0x19b
  github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).analyzeSelectList()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/project.go:160 +0x37b
  github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).analyzeProjectionList()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/project.go:94 +0x3a7
  github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildSelectClause()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:1012 +0x415
  github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildSelectStmtWithoutParens()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:949 +0x26e
  github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildSelect.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:922 +0x14c
  github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).processWiths()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/with.go:115 +0xcf
  github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildSelect()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/select.go:921 +0x71c
  github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildStmt()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:302 +0x49e
  github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildStmtAtRoot()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:252 +0x2e8
  github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).Build()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:226 +0x596
  github.com/cockroachdb/cockroach/pkg/sql.(*instrumentationHelper).SetIndexRecommendations()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/instrumentation.go:709 +0x74f
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1201 +0x1a7c
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:699 +0x2e06
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:148 +0x972
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:236 +0x1f4
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2021 +0xaa8
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2023 +0x7a6
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1869 +0x415
  github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:212 +0x114

Previous read at 0x00c001287ee8 by goroutine 1119:
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*CustomFuncs).EqualsNumber()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/general_funcs.go:990 +0x104
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).ConstructMinus()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.og.go:18120 +0x4bc
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).CopyAndReplaceDefault()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.og.go:24219 +0x488c
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).AssignPlaceholders.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.go:313 +0x14b
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).invokeReplace()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.og.go:24959 +0x8a
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).CopyAndReplaceDefault()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.og.go:24323 +0x50ed
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).AssignPlaceholders.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.go:313 +0x14b
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).invokeReplace()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.og.go:24959 +0x8a
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).CopyAndReplaceDefault()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.og.go:24227 +0x5768
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).AssignPlaceholders.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.go:313 +0x14b
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).invokeReplace()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.og.go:24959 +0x8a
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).CopyAndReplaceDefault()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.og.go:24045 +0xe7ad
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).AssignPlaceholders.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.go:313 +0x14b
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).invokeReplace()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.og.go:24959 +0x8a
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).CopyAndReplaceDefault()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.og.go:24342 +0x4a3a
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).AssignPlaceholders.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.go:313 +0x14b
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).invokeReplace()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.og.go:24959 +0x8a
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).copyAndReplaceDefaultScalarListExpr()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.og.go:24948 +0xc4
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).CopyAndReplaceDefault()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.og.go:24373 +0x88f5
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).AssignPlaceholders.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.go:313 +0x14b
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).invokeReplace()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.og.go:24959 +0x8a
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).CopyAndReplaceDefault()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.og.go:24323 +0x50ed
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).AssignPlaceholders.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.go:313 +0x14b
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).invokeReplace()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.og.go:24959 +0x8a
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).CopyAndReplaceDefault()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.og.go:24215 +0xb468
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).AssignPlaceholders.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.go:313 +0x14b
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).invokeReplace()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.og.go:24959 +0x8a
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).copyAndReplaceDefaultProjectionsExpr()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.og.go:24883 +0x1ac
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).CopyAndReplaceDefault()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.og.go:23655 +0xbdc
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).AssignPlaceholders.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.go:313 +0x14b
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).invokeReplace()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.og.go:24959 +0x8a
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).CopyAndReplace()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.go:272 +0x218
  github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).AssignPlaceholders()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.go:315 +0x178
  github.com/cockroachdb/cockroach/pkg/sql.(*optPlanningCtx).reuseMemo()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:485 +0xc4
  github.com/cockroachdb/cockroach/pkg/sql.(*optPlanningCtx).buildExecMemo()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:518 +0x2be
  github.com/cockroachdb/cockroach/pkg/sql.(*planner).makeOptimizerPlan()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:228 +0x1c4
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).makeExecPlan()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1409 +0x72
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1054 +0x3f7
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:699 +0x2e06
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:148 +0x972
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:236 +0x1f4
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2021 +0xaa8
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2023 +0x7a6
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1869 +0x415
  github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:212 +0x114

Goroutine 2068 (running) created at:
  github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:211 +0xaa6
  github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:743 +0xbe9
  github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryIteratorEx()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:578 +0x2a5
  github.com/cockroachdb/cockroach/pkg/sql.glob..func31()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/crdb_internal.go:818 +0x436
  github.com/cockroachdb/cockroach/pkg/sql.(*virtualDefEntry).getPlanInfo.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/virtual_schema.go:562 +0xbf8
  github.com/cockroachdb/cockroach/pkg/sql.constructVirtualScan.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/exec_factory_util.go:250 +0x7e
  github.com/cockroachdb/cockroach/pkg/sql.(*delayedNode).startExec()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/delayed.go:50 +0xbb
  github.com/cockroachdb/cockroach/pkg/sql.startExec.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/plan.go:517 +0x183
  github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:112 +0xa2
  runtime.deferreturn()
      /usr/local/go/src/runtime/panic.go:436 +0x32
  github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:79 +0x135
  github.com/cockroachdb/cockroach/pkg/sql.walkPlan()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:43 +0x158
  github.com/cockroachdb/cockroach/pkg/sql.startExec()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/plan.go:520 +0x170
  github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:147 +0x124
  github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Columnarizer).Init()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/columnarizer.go:162 +0x14b
  github.com/cockroachdb/cockroach/pkg/sql/colexec.(*invariantsChecker).Init()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/invariants_checker.go:67 +0x97
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Init()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:90 +0x58
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedStatsCollectorImpl).Init()
      <autogenerated>:1 +0x4b
  github.com/cockroachdb/cockroach/pkg/sql/colexecop.(*OneInputHelper).Init()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecop/operator.go:296 +0x97
  github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecsel.(*selRegexpBytesBytesConstOp).Init()
      <autogenerated>:1 +0x4b
  github.com/cockroachdb/cockroach/pkg/sql/colexecop.(*OneInputInitCloserHelper).Init()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecop/operator.go:375 +0x93
  github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*simpleProjectOp).Init()
      <autogenerated>:1 +0x4f
  github.com/cockroachdb/cockroach/pkg/sql/colexec.(*invariantsChecker).Init()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/invariants_checker.go:67 +0x97
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Init()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:90 +0x58
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedStatsCollectorImpl).Init()
      <autogenerated>:1 +0x4b
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).init.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:237 +0x68
  github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91 +0x75
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).init()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:236 +0x70
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:281 +0x2eb
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:319 +0x31e
  github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:698 +0xfd6
  github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1568 +0x3a4
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1537 +0xcf4
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1172 +0x1446
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:699 +0x2e06
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:148 +0x972
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1943 +0x717
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1947 +0x132f
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1869 +0x415
  github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:808 +0x1bd
  github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:728 +0x539

Goroutine 1119 (running) created at:
  github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:211 +0xaa6
  github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:743 +0xbe9
  github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryIteratorEx()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:578 +0x2a5
  github.com/cockroachdb/cockroach/pkg/sql.glob..func31()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/crdb_internal.go:818 +0x436
  github.com/cockroachdb/cockroach/pkg/sql.(*virtualDefEntry).getPlanInfo.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/virtual_schema.go:562 +0xbf8
  github.com/cockroachdb/cockroach/pkg/sql.constructVirtualScan.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/exec_factory_util.go:250 +0x7e
  github.com/cockroachdb/cockroach/pkg/sql.(*delayedNode).startExec()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/delayed.go:50 +0xbb
  github.com/cockroachdb/cockroach/pkg/sql.startExec.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/plan.go:517 +0x183
  github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:112 +0xa2
  runtime.deferreturn()
      /usr/local/go/src/runtime/panic.go:436 +0x32
  github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:79 +0x135
  github.com/cockroachdb/cockroach/pkg/sql.walkPlan()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:43 +0x158
  github.com/cockroachdb/cockroach/pkg/sql.startExec()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/plan.go:520 +0x170
  github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:147 +0x124
  github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Columnarizer).Init()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/columnarizer.go:162 +0x14b
  github.com/cockroachdb/cockroach/pkg/sql/colexec.(*invariantsChecker).Init()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/invariants_checker.go:67 +0x97
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Init()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:90 +0x58
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedStatsCollectorImpl).Init()
      <autogenerated>:1 +0x4b
  github.com/cockroachdb/cockroach/pkg/sql/colexecop.(*OneInputHelper).Init()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecop/operator.go:296 +0x97
  github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecsel.(*selRegexpBytesBytesConstOp).Init()
      <autogenerated>:1 +0x4b
  github.com/cockroachdb/cockroach/pkg/sql/colexecop.(*OneInputInitCloserHelper).Init()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecop/operator.go:375 +0x93
  github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*simpleProjectOp).Init()
      <autogenerated>:1 +0x4f
  github.com/cockroachdb/cockroach/pkg/sql/colexec.(*invariantsChecker).Init()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/invariants_checker.go:67 +0x97
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Init()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:90 +0x58
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedStatsCollectorImpl).Init()
      <autogenerated>:1 +0x4b
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).init.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:237 +0x68
  github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91 +0x75
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).init()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:236 +0x70
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:281 +0x2eb
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:319 +0x31e
  github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:698 +0xfd6
  github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1568 +0x3a4
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1537 +0xcf4
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1172 +0x1446
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:699 +0x2e06
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:148 +0x972
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1943 +0x717
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1947 +0x132f
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1869 +0x415
  github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:808 +0x1bd
  github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:728 +0x539
==================
FAIL	github.com/cockroachdb/cockroach/pkg/sql	3413.246s

Jira issue: CRDB-18493

yuzefovich avatar Aug 10 '22 23:08 yuzefovich

note [github.com/cockroachdb/cockroach/pkg/sql.(*instrumentationHelper).SetIndexRecommendations()](http://github.com/cockroachdb/cockroach/pkg/sql.(*instrumentationHelper).SetIndexRecommendations())

ajwerner avatar Aug 10 '22 23:08 ajwerner

Indeed, the index recommendations are what introduced the data race, but I think the problem is that we're modifying the AST when we didn't expect to.

yuzefovich avatar Aug 11 '22 02:08 yuzefovich

So my understanding is that this race always occurs:

  • when executing portals
  • with two concurrent internal queries
  • one of the goroutines is setting the index recommendations.

The problem appears to be that we are reusing the same memo that is modified during optbuilder.Builder.Build - is that expected that we're modifying the reused memo? cc @mgartner

Also, do we want to generate the index recommendations for the internal queries? cc @maryliag

yuzefovich avatar Aug 11 '22 02:08 yuzefovich

That's a good point, I created a PR to ignore recommendations for internal queries.

maryliag avatar Aug 11 '22 12:08 maryliag