cockroach icon indicating copy to clipboard operation
cockroach copied to clipboard

Published 20 hours ago • verified publisher icon cockroachdb

opt: v23.1.12: span must be fully contained in the bucket

Open cockroach-sentry opened this issue 1 year ago • 3 comments

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/5351250767/?referrer=webhooks_plugin

Panic Message:

histogram.go:694: span must be fully contained in the bucket
(1) assertion failure
Wraps: (2) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/sql/opt/props.getFilteredBucket
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/props/histogram.go:694
  | github.com/cockroachdb/cockroach/pkg/sql/opt/props.(*Histogram).filter
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/props/histogram.go:341
  | github.com/cockroachdb/cockroach/pkg/sql/opt/props.(*Histogram).Filter
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/props/histogram.go:407
  | github.com/cockroachdb/cockroach/pkg/sql/opt/memo.(*statisticsBuilder).updateHistogram
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/memo/statistics_builder.go:3637
  | github.com/cockroachdb/cockroach/pkg/sql/opt/memo.(*statisticsBuilder).applyIndexConstraint.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/memo/statistics_builder.go:3401
  | github.com/cockroachdb/cockroach/pkg/sql/opt.ColSet.ForEach.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/colset.go:83
  | github.com/cockroachdb/cockroach/pkg/util/intsets.Fast.ForEach
  | 	github.com/cockroachdb/cockroach/pkg/util/intsets/fast.go:154
  | github.com/cockroachdb/cockroach/pkg/sql/opt.ColSet.ForEach
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/colset.go:83
  | github.com/cockroachdb/cockroach/pkg/sql/opt/memo.(*statisticsBuilder).applyIndexConstraint
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/memo/statistics_builder.go:3399
  | github.com/cockroachdb/cockroach/pkg/sql/opt/memo.(*statisticsBuilder).constrainScan
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/memo/statistics_builder.go:948
  | github.com/cockroachdb/cockroach/pkg/sql/opt/memo.(*statisticsBuilder).buildScan
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/memo/statistics_builder.go:859
  | github.com/cockroachdb/cockroach/pkg/sql/opt/memo.(*logicalPropsBuilder).buildScanProps
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/memo/logical_props_builder.go:186
  | github.com/cockroachdb/cockroach/pkg/sql/opt/memo.(*Memo).MemoizeScan
  | 	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/opt/memo/expr.og.go:20146
  | github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).ConstructScan
  | 	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/opt/norm/factory.og.go:456
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*indexScanBuilder).Build
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/index_scan_builder.go:249
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*CustomFuncs).GenerateConstrainedScans.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/select_funcs.go:505
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*scanIndexIter).ForEachStartingAfter
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/scan_index_iter.go:305
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*scanIndexIter).ForEach
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/scan_index_iter.go:208
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*CustomFuncs).GenerateConstrainedScans
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/select_funcs.go:441
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*explorer).exploreSelect
  | 	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/opt/xform/explorer.og.go:259
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*explorer).exploreGroupMember
  | 	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/opt/xform/explorer.og.go:24
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*explorer).exploreGroup
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/explorer.go:185
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*Optimizer).optimizeGroup
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:536
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*Optimizer).optimizeEnforcer
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:717
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*Optimizer).enforceProps
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:668
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*Optimizer).optimizeGroupMember
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:563
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*Optimizer).optimizeGroup
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:523
  | github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*Optimizer).Optimize
  | 	github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:269
  | github.com/cockroachdb/cockroach/pkg/sql.(*optPlanningCtx).reuseMemo
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:507
  | github.com/cockroachdb/cockroach/pkg/sql.(*optPlanningCtx).buildExecMemo
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:537
  | github.com/cockroachdb/cockroach/pkg/sql.(*planner).makeOptimizerPlan
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:240
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).makeExecPlan
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2028
Wraps: (3) span must be fully contained in the bucket
Error types: (1) *assert.withAssertionFailure (2) *withstack.withStack (3) *errutil.leafError
-- report composition:
*errutil.leafError: span must be fully contained in the bucket
histogram.go:694: *withstack.withStack (top exception)
*assert.withAssertionFailure
Stacktrace (expand for inline code snippets):

https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/conn_executor_exec.go#L2027-L2029 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/plan_opt.go#L239-L241 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/plan_opt.go#L536-L538 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/plan_opt.go#L506-L508 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/opt/xform/optimizer.go#L268-L270 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/opt/xform/optimizer.go#L522-L524 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/opt/xform/optimizer.go#L562-L564 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/opt/xform/optimizer.go#L667-L669 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/opt/xform/optimizer.go#L716-L718 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/opt/xform/optimizer.go#L535-L537 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/opt/xform/explorer.go#L184-L186 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/bazel-out/k8-opt/bin/pkg/sql/opt/xform/explorer.og.go#L23-L25 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/bazel-out/k8-opt/bin/pkg/sql/opt/xform/explorer.og.go#L258-L260 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/opt/xform/select_funcs.go#L440-L442 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/opt/xform/scan_index_iter.go#L207-L209 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/opt/xform/scan_index_iter.go#L304-L306 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/opt/xform/select_funcs.go#L504-L506 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/opt/xform/index_scan_builder.go#L248-L250 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/bazel-out/k8-opt/bin/pkg/sql/opt/norm/factory.og.go#L455-L457 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/bazel-out/k8-opt/bin/pkg/sql/opt/memo/expr.og.go#L20145-L20147 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/opt/memo/logical_props_builder.go#L185-L187 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/opt/memo/statistics_builder.go#L858-L860 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/opt/memo/statistics_builder.go#L947-L949 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/opt/memo/statistics_builder.go#L3398-L3400 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/opt/colset.go#L82-L84 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/util/intsets/fast.go#L153-L155 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/opt/colset.go#L82-L84 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/opt/memo/statistics_builder.go#L3400-L3402 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/opt/memo/statistics_builder.go#L3636-L3638 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/opt/props/histogram.go#L406-L408 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/opt/props/histogram.go#L340-L342 https://github.com/cockroachdb/cockroach/blob/d7e9824b4cd6ebf7a8548156f2a772ae6648257d/pkg/sql/opt/props/histogram.go#L693-L695

pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).makeExecPlan at line 2028
pkg/sql/plan_opt.go in pkg/sql.(*planner).makeOptimizerPlan at line 240
pkg/sql/plan_opt.go in pkg/sql.(*optPlanningCtx).buildExecMemo at line 537
pkg/sql/plan_opt.go in pkg/sql.(*optPlanningCtx).reuseMemo at line 507
pkg/sql/opt/xform/optimizer.go in pkg/sql/opt/xform.(*Optimizer).Optimize at line 269
pkg/sql/opt/xform/optimizer.go in pkg/sql/opt/xform.(*Optimizer).optimizeGroup at line 523
pkg/sql/opt/xform/optimizer.go in pkg/sql/opt/xform.(*Optimizer).optimizeGroupMember at line 563
pkg/sql/opt/xform/optimizer.go in pkg/sql/opt/xform.(*Optimizer).enforceProps at line 668
pkg/sql/opt/xform/optimizer.go in pkg/sql/opt/xform.(*Optimizer).optimizeEnforcer at line 717
pkg/sql/opt/xform/optimizer.go in pkg/sql/opt/xform.(*Optimizer).optimizeGroup at line 536
pkg/sql/opt/xform/explorer.go in pkg/sql/opt/xform.(*explorer).exploreGroup at line 185
bazel-out/k8-opt/bin/pkg/sql/opt/xform/explorer.og.go in pkg/sql/opt/xform.(*explorer).exploreGroupMember at line 24
bazel-out/k8-opt/bin/pkg/sql/opt/xform/explorer.og.go in pkg/sql/opt/xform.(*explorer).exploreSelect at line 259
pkg/sql/opt/xform/select_funcs.go in pkg/sql/opt/xform.(*CustomFuncs).GenerateConstrainedScans at line 441
pkg/sql/opt/xform/scan_index_iter.go in pkg/sql/opt/xform.(*scanIndexIter).ForEach at line 208
pkg/sql/opt/xform/scan_index_iter.go in pkg/sql/opt/xform.(*scanIndexIter).ForEachStartingAfter at line 305
pkg/sql/opt/xform/select_funcs.go in pkg/sql/opt/xform.(*CustomFuncs).GenerateConstrainedScans.func1 at line 505
pkg/sql/opt/xform/index_scan_builder.go in pkg/sql/opt/xform.(*indexScanBuilder).Build at line 249
bazel-out/k8-opt/bin/pkg/sql/opt/norm/factory.og.go in pkg/sql/opt/norm.(*Factory).ConstructScan at line 456
bazel-out/k8-opt/bin/pkg/sql/opt/memo/expr.og.go in pkg/sql/opt/memo.(*Memo).MemoizeScan at line 20146
pkg/sql/opt/memo/logical_props_builder.go in pkg/sql/opt/memo.(*logicalPropsBuilder).buildScanProps at line 186
pkg/sql/opt/memo/statistics_builder.go in pkg/sql/opt/memo.(*statisticsBuilder).buildScan at line 859
pkg/sql/opt/memo/statistics_builder.go in pkg/sql/opt/memo.(*statisticsBuilder).constrainScan at line 948
pkg/sql/opt/memo/statistics_builder.go in pkg/sql/opt/memo.(*statisticsBuilder).applyIndexConstraint at line 3399
pkg/sql/opt/colset.go in pkg/sql/opt.ColSet.ForEach at line 83
pkg/util/intsets/fast.go in pkg/util/intsets.Fast.ForEach at line 154
pkg/sql/opt/colset.go in pkg/sql/opt.ColSet.ForEach.func1 at line 83
pkg/sql/opt/memo/statistics_builder.go in pkg/sql/opt/memo.(*statisticsBuilder).applyIndexConstraint.func1 at line 3401
pkg/sql/opt/memo/statistics_builder.go in pkg/sql/opt/memo.(*statisticsBuilder).updateHistogram at line 3637
pkg/sql/opt/props/histogram.go in pkg/sql/opt/props.(*Histogram).Filter at line 407
pkg/sql/opt/props/histogram.go in pkg/sql/opt/props.(*Histogram).filter at line 341
pkg/sql/opt/props/histogram.go in pkg/sql/opt/props.getFilteredBucket at line 694

Tags

Tag Value
Command server
Environment v23.1.12
Go Version go1.19.13
Platform linux amd64
Distribution CCL
Cockroach Release v23.1.12
Cockroach SHA d7e9824b4cd6ebf7a8548156f2a772ae6648257d
# of CPUs 16
# of Goroutines 6176

Jira issue: CRDB-38748

cockroach-sentry avatar May 14 '24 21:05 cockroach-sentry

dup of https://github.com/cockroachlabs/support/issues/2952

yuzefovich avatar May 20 '24 17:05 yuzefovich

I'm going to re-open this to use it as the public tracking issue.

Here's how to reproduce the problem:

  1. Start a multi-region demo cluster: 
    ./cockroach demo \
--nodes 12 \
--no-example-database \
--insecure \
--multitenant=false \
--demo-locality=\
cloud=aws,region=ap-northeast-2,zone=ap-northeast-2b,pg=ap-northeast-2b1:\
cloud=aws,region=ap-northeast-2,zone=ap-northeast-2a,pg=ap-northeast-2a1:\
cloud=aws,region=ap-northeast-2,zone=ap-northeast-2c,pg=ap-northeast-2c1:\
cloud=aws,region=eu-west-1,zone=eu-west-1b,pg=eu-west-1b1:\
cloud=aws,region=eu-west-1,zone=eu-west-1a,pg=eu-west-1a1:\
cloud=aws,region=us-east-1,zone=us-east-1c,pg=us-east-1c1:\
cloud=aws,region=eu-central-1,zone=eu-central-1b,pg=eu-central-1b1:\
cloud=aws,region=eu-central-1,zone=eu-central-1a,pg=eu-central-1a1:\
cloud=aws,region=eu-central-1,zone=eu-central-1c,pg=eu-central-1c1:\
cloud=aws,region=us-east-1,zone=us-east-1a,pg=us-east-1a1:\
cloud=aws,region=us-east-1,zone=us-east-1b,pg=us-east-1b2:\
cloud=aws,region=eu-west-1,zone=eu-west-1c,pg=eu-west-1c2
  2. Turn off autostats: 
    SET CLUSTER SETTING sql.stats.automatic_collection.enabled = off;
  3. Create this REGIONAL BY ROW table: 
    CREATE DATABASE db PRIMARY REGION "us-east-1" REGIONS = "ap-northeast-2", "eu-west-1", "us-east-1" SURVIVE REGION FAILURE;
USE db;

CREATE TABLE t (
  region crdb_internal_region NOT NULL,
  id UUID NOT NULL DEFAULT gen_random_uuid(),
  a INT NOT NULL,
  PRIMARY KEY (id),
  UNIQUE INDEX (a)
) LOCALITY REGIONAL BY ROW AS region;

INSERT INTO t (region, a) VALUES ('ap-northeast-2', 0), ('eu-west-1', 1), ('us-east-1', 2);

ANALYZE t;
  4. Add another region to the database: 
    ALTER DATABASE db ADD REGION "eu-central-1";
  5. Now this statement will fail with an internal error: 
    INSERT INTO t (region, a) VALUES ('eu-west-1', 3);

michae2 avatar May 22 '24 23:05 michae2

This is related to https://github.com/cockroachdb/cockroach/issues/67050.

michae2 avatar May 22 '24 23:05 michae2