v0.2.0 fail to work on multiple cases [ panic: runtime error: index out of range ]

[fruch]

Packages

Scylla version: 6.2.3-20250119.bff9ddde1283 with build-id 1d62472ad27cbcd54a8de19cc9171ebf3cde0af7

Kernel Version: 6.8.0-1021-aws

Issue description

• [x] This issue is a regression.
• [ ] It is unknown if this issue is a regression.

s-b panicing in the middle of test

panic: runtime error: index out of range [-1]

goroutine 142 [running]:
math/rand.(*rngSource).Uint64(...)
	/opt/hostedtoolcache/go/1.24.1/x64/src/math/rand/rng.go:249
math/rand.(*rngSource).Int63(0x8a4c80?)
	/opt/hostedtoolcache/go/1.24.1/x64/src/math/rand/rng.go:234 +0x92
github.com/scylladb/scylla-bench/random.(*lockedSource).Int63(0xc00008e870)
	/home/runner/work/scylla-bench/scylla-bench/random/random.go:291 +0x46
math/rand.(*Rand).Int63(...)
	/opt/hostedtoolcache/go/1.24.1/x64/src/math/rand/rand.go:96
math/rand.(*Rand).Int63n(0xc00008e8a0, 0x39c)
	/opt/hostedtoolcache/go/1.24.1/x64/src/math/rand/rand.go:128 +0x5e
github.com/scylladb/scylla-bench/random.Uniform.Generate(...)
	/home/runner/work/scylla-bench/scylla-bench/random/random.go:281
main.DoBatchedWrites.func1(0xc0001381e0)
	/home/runner/work/scylla-bench/scylla-bench/modes.go:422 +0x16a
main.RunTest(0xc0001381e0, {0x98cf88, 0xc000206000}, {0x989608, 0xca2820}, 0xc000241ea0)
	/home/runner/work/scylla-bench/scylla-bench/modes.go:164 +0x174
main.DoBatchedWrites(0xc000192008, 0xc0001381e0, {0x98cf88, 0xc000206000}, {0x989608, 0xca2820})
	/home/runner/work/scylla-bench/scylla-bench/modes.go:405 +0x11e
main.main.func4(0x0, 0xc0001381e0, {0x989608, 0xca2820})
	/home/runner/work/scylla-bench/scylla-bench/main.go:658 +0xee
main.RunConcurrently.func1(0x0)
	/home/runner/work/scylla-bench/scylla-bench/modes.go:91 +0x5d
created by main.RunConcurrently in goroutine 1
	/home/runner/work/scylla-bench/scylla-bench/modes.go:89 +0x1a7

Impact

it blocks multiple cases

How frequently does it reproduce?

multiple cases

Installation details

Cluster size: 4 nodes (i4i.2xlarge)

Scylla Nodes used in this run:

• rolling-upgrade--ubuntu-focal-db-node-aaf7b571-4 (52.18.211.5 | 10.4.0.216) (shards: 7)
• rolling-upgrade--ubuntu-focal-db-node-aaf7b571-3 (34.252.205.176 | 10.4.2.216) (shards: 7)
• rolling-upgrade--ubuntu-focal-db-node-aaf7b571-2 (34.247.183.151 | 10.4.3.250) (shards: 7)
• rolling-upgrade--ubuntu-focal-db-node-aaf7b571-1 (54.154.109.11 | 10.4.2.122) (shards: 7)

OS / Image: ami-062faa721a0d5b39c (aws: undefined_region)

Test: rolling-upgrade-ami-test Test id: aaf7b571-6d4b-4a1d-83eb-e2b327120f31 Test name: scylla-master/reproducers/rolling-upgrade-ami-test Test method: upgrade_test.UpgradeTest.test_rolling_upgrade Test config file(s):

• rolling-upgrade.yaml

Logs and commands

• Restore Monitor Stack command: $ hydra investigate show-monitor aaf7b571-6d4b-4a1d-83eb-e2b327120f31
• Restore monitor on AWS instance using Jenkins job
• Show all stored logs command: $ hydra investigate show-logs aaf7b571-6d4b-4a1d-83eb-e2b327120f31

Logs:

• db-cluster-aaf7b571.tar.zst - https://cloudius-jenkins-test.s3.amazonaws.com/aaf7b571-6d4b-4a1d-83eb-e2b327120f31/20250324_095016/db-cluster-aaf7b571.tar.zst
• sct-runner-events-aaf7b571.tar.zst - https://cloudius-jenkins-test.s3.amazonaws.com/aaf7b571-6d4b-4a1d-83eb-e2b327120f31/20250324_095016/sct-runner-events-aaf7b571.tar.zst
• sct-aaf7b571.log.tar.zst - https://cloudius-jenkins-test.s3.amazonaws.com/aaf7b571-6d4b-4a1d-83eb-e2b327120f31/20250324_095016/sct-aaf7b571.log.tar.zst
• loader-set-aaf7b571.tar.zst - https://cloudius-jenkins-test.s3.amazonaws.com/aaf7b571-6d4b-4a1d-83eb-e2b327120f31/20250324_095016/loader-set-aaf7b571.tar.zst
• monitor-set-aaf7b571.tar.zst - https://cloudius-jenkins-test.s3.amazonaws.com/aaf7b571-6d4b-4a1d-83eb-e2b327120f31/20250324_095016/monitor-set-aaf7b571.tar.zst
• ssl-conf-aaf7b571.tar.zst - https://cloudius-jenkins-test.s3.amazonaws.com/aaf7b571-6d4b-4a1d-83eb-e2b327120f31/20250324_095016/ssl-conf-aaf7b571.tar.zst
• builder-aaf7b571.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/aaf7b571-6d4b-4a1d-83eb-e2b327120f31/upload_20250324_095144/builder-aaf7b571.log.tar.gz

Jenkins job URL Argus

[timtimb0t]

reproduced there: https://argus.scylladb.com/tests/scylla-cluster-tests/1e2639c4-dcef-4360-b3e3-a2e910f3b759

[fruch]

@CodeLieutenant

please look at this ASAP, this version was backported into 2025.1 and it breaking everywhere s-b is used

[CodeLieutenant]

@CodeLieutenant

please look at this ASAP, this version was backported into 2025.1 and it breaking everywhere s-b is used

Working on it already

[timtimb0t]

reproduced again https://argus.scylladb.com/tests/scylla-cluster-tests/73e51a36-df58-4132-bf21-2a3889b1e6d2