cockroach roachtest: cluster

roachtest.cluster_creation failed with artifacts on master @ 826ee929e41e2d1a1634a18348a8495c5ec5218d:

test tpcc-nowait/literal/w=1000/nodes=5/cpu=16 failed: (test_runner.go:850).func4: in provider: ibm: failed to create instance teamcity-19868876-1749184215-504-n6cpu16-0004 in region ca-tor: instance teamcity-19868876-1749184215-504-n6cpu16-0004 didn't reach stable state in 2m30s; [instance state: starting|network attached: false]
failed to create instance teamcity-19868876-1749184215-504-n6cpu16-0002 in region ca-tor: instance teamcity-19868876-1749184215-504-n6cpu16-0002 didn't reach stable state in 2m30s; [instance state: starting|network attached: false]
failed to create instance teamcity-19868876-1749184215-504-n6cpu16-0001 in region ca-tor: instance teamcity-19868876-1749184215-504-n6cpu16-0001 didn't reach stable state in 2m30s; [instance state: starting|network attached: false]
failed to create instance teamcity-19868876-1749184215-504-n6cpu16-0005 in region ca-tor: instance teamcity-19868876-1749184215-504-n6cpu16-0005 didn't reach stable state in 2m30s; [instance state: starting|network attached: false]
failed to create instance teamcity-19868876-1749184215-504-n6cpu16-0003 in region ca-tor: instance teamcity-19868876-1749184215-504-n6cpu16-0003 didn't reach stable state in 2m30s; [instance state: starting|network attached: false] [owner=test-eng]

Parameters:

cloud=ibm
coverageBuild=false
cpu=16
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/test-eng _{This test on roachdash | Improve this report!}

Jira issue: CRDB-51305

Jun 06 '25 10:06 cockroach-teamcity

roachtest.cluster_creation failed with artifacts on master @ 826ee929e41e2d1a1634a18348a8495c5ec5218d:

test ycsb/B/nodes=3 failed: (test_runner.go:850).func4: in provider: ibm: failed to create instance teamcity-19868876-1749184215-510-n4cpu8-0001 in region ca-tor: instance teamcity-19868876-1749184215-510-n4cpu8-0001 didn't reach stable state in 2m0s; [instance state: starting|network attached: false] [owner=test-eng]

Parameters:

cloud=ibm
coverageBuild=false
cpu=8
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 06 '25 10:06 cockroach-teamcity

roachtest.cluster_creation failed with artifacts on master @ 826ee929e41e2d1a1634a18348a8495c5ec5218d:

test ycsb/A/nodes=3 failed: (test_runner.go:850).func4: in provider: ibm: failed to create instance teamcity-19868876-1749184215-508-n4cpu8-0001 in region ca-tor: instance teamcity-19868876-1749184215-508-n4cpu8-0001 didn't reach stable state in 2m0s; [instance state: starting|network attached: false]
failed to create instance teamcity-19868876-1749184215-508-n4cpu8-0003 in region ca-tor: instance teamcity-19868876-1749184215-508-n4cpu8-0003 didn't reach stable state in 2m0s; [instance state: starting|network attached: false] [owner=test-eng]

Parameters:

cloud=ibm
coverageBuild=false
cpu=8
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 06 '25 10:06 cockroach-teamcity

roachtest.cluster_creation failed with artifacts on master @ 826ee929e41e2d1a1634a18348a8495c5ec5218d:

test ycsb/F/nodes=3 failed: (test_runner.go:850).func4: in provider: ibm: failed to create instance teamcity-19868876-1749184215-513-n4cpu8-0003 in region ca-tor: instance teamcity-19868876-1749184215-513-n4cpu8-0003 didn't reach stable state in 2m0s; [instance state: starting|network attached: false] [owner=test-eng]

Parameters:

cloud=ibm
coverageBuild=false
cpu=8
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 06 '25 10:06 cockroach-teamcity

roachtest.cluster_creation failed with artifacts on master @ 826ee929e41e2d1a1634a18348a8495c5ec5218d:

test ycsb/E/nodes=3 failed: (test_runner.go:850).func4: in provider: ibm: failed to create instance teamcity-19868876-1749184215-514-n4cpu8-0003 in region ca-tor: instance teamcity-19868876-1749184215-514-n4cpu8-0003 didn't reach stable state in 2m0s; [instance state: starting|network attached: false] [owner=test-eng]

Parameters:

cloud=ibm
coverageBuild=false
cpu=8
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 06 '25 10:06 cockroach-teamcity

roachtest.cluster_creation failed with artifacts on master @ 44f6c3d374d86abece1e3c19fd0b0bbc7d68b7c7:

test kv95/enc=false/nodes=3/cpu=32/size=64kb failed: (test_runner.go:850).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-RqfkpYn5udtCJffO0YGJeroW-lg= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19876411-1749274909-34-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-06-07T06:35:34Z},{Key=Name,Value=teamcity-19876411-1749274909-34-n4cpu32-0002},{Key=Cluster,Value=teamcity-19876411-1749274909-34-n4cpu32}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19876411-1749274909-34-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-06-07T06:35:34Z},{Key=Name,Value=teamcity-19876411-1749274909-34-n4cpu32-0002},{Key=Cluster,Value=teamcity-19876411-1749274909-34-n4cpu32}] --user-data file:///tmp/aws-startup-script3757608981 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2574264663 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c7gd.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c7gd.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

cloud=aws
coverageBuild=false
cpu=32
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 07 '25 06:06 cockroach-teamcity

roachtest.cluster_creation failed with artifacts on master @ 44f6c3d374d86abece1e3c19fd0b0bbc7d68b7c7:

test ycsb/B/nodes=3/cpu=32/uniform failed: (test_runner.go:850).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-RqfkpYn5udtCJffO0YGJeroW-lg= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-06-07T07:12:07Z},{Key=Name,Value=teamcity-19876411-1749274909-52-n4cpu32-0004},{Key=Cluster,Value=teamcity-19876411-1749274909-52-n4cpu32},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19876411-1749274909-52-n4cpu32}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-06-07T07:12:07Z},{Key=Name,Value=teamcity-19876411-1749274909-52-n4cpu32-0004},{Key=Cluster,Value=teamcity-19876411-1749274909-52-n4cpu32},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19876411-1749274909-52-n4cpu32}] --user-data file:///tmp/aws-startup-script4273110941 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3777221438 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

cloud=aws
coverageBuild=false
cpu=32
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 07 '25 07:06 cockroach-teamcity

roachtest.cluster_creation failed with artifacts on master @ 44f6c3d374d86abece1e3c19fd0b0bbc7d68b7c7:

test costfuzz/rand-multi-region failed: (test_runner.go:850).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

cloud=azure
coverageBuild=false
cpu=4
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 07 '25 07:06 cockroach-teamcity

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation failed with artifacts on master @ 44f6c3d374d86abece1e3c19fd0b0bbc7d68b7c7:

test costfuzz/seed-multi-region failed: (test_runner.go:850).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

cloud=azure
coverageBuild=false
cpu=4
runtimeAssertionsBuild=true
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 07 '25 07:06 cockroach-teamcity

roachtest.cluster_creation failed with artifacts on master @ 44f6c3d374d86abece1e3c19fd0b0bbc7d68b7c7:

test ycsb/D/nodes=3/cpu=32/isolation-level=read-committed failed: (test_runner.go:850).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-RqfkpYn5udtCJffO0YGJeroW-lg= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=created,Value=2025-06-07T07:19:06Z},{Key=Name,Value=teamcity-19876411-1749274909-56-n4cpu32-0002},{Key=Cluster,Value=teamcity-19876411-1749274909-56-n4cpu32},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19876411-1749274909-56-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=created,Value=2025-06-07T07:19:06Z},{Key=Name,Value=teamcity-19876411-1749274909-56-n4cpu32-0002},{Key=Cluster,Value=teamcity-19876411-1749274909-56-n4cpu32},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19876411-1749274909-56-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64}] --user-data file:///tmp/aws-startup-script628708739 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping753103623 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c7gd.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c7gd.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

cloud=aws
coverageBuild=false
cpu=32
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 07 '25 07:06 cockroach-teamcity

roachtest.cluster_creation failed with artifacts on master @ 44f6c3d374d86abece1e3c19fd0b0bbc7d68b7c7:

test ycsb/D/nodes=3/cpu=32/mvcc-range-keys=global failed: (test_runner.go:850).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-RqfkpYn5udtCJffO0YGJeroW-lg= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-19876411-1749274909-59-n4cpu32-0002},{Key=Cluster,Value=teamcity-19876411-1749274909-59-n4cpu32},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19876411-1749274909-59-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-06-07T07:20:48Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-19876411-1749274909-59-n4cpu32-0002},{Key=Cluster,Value=teamcity-19876411-1749274909-59-n4cpu32},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19876411-1749274909-59-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-06-07T07:20:48Z}] --user-data file:///tmp/aws-startup-script1849498756 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping420860778 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

cloud=aws
coverageBuild=false
cpu=32
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 07 '25 07:06 cockroach-teamcity

roachtest.cluster_creation failed with artifacts on master @ 44f6c3d374d86abece1e3c19fd0b0bbc7d68b7c7:

test ycsb/E/nodes=3/cpu=32 failed: (test_runner.go:850).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-RqfkpYn5udtCJffO0YGJeroW-lg= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19876411-1749274909-63-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-06-07T07:25:39Z},{Key=Name,Value=teamcity-19876411-1749274909-63-n4cpu32-0003},{Key=Cluster,Value=teamcity-19876411-1749274909-63-n4cpu32},{Key=spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19876411-1749274909-63-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-06-07T07:25:39Z},{Key=Name,Value=teamcity-19876411-1749274909-63-n4cpu32-0003},{Key=Cluster,Value=teamcity-19876411-1749274909-63-n4cpu32},{Key=spot,Value=true}] --user-data file:///tmp/aws-startup-script963008117 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3994773850 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

cloud=aws
coverageBuild=false
cpu=32
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 07 '25 07:06 cockroach-teamcity

roachtest.cluster_creation failed with artifacts on master @ 44f6c3d374d86abece1e3c19fd0b0bbc7d68b7c7:

test clearrange/zfs/checks=true failed: (test_runner.go:850).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --format json --labels usage=roachtest,arch=arm64,created=2025-06-07t08_11_53z,roachprod=true,cluster=teamcity-19876410-1749274874-49-n10cpu16,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20240319 --image-project ubuntu-os-cloud --boot-disk-type  --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-16 --metadata-from-file startup-script=/tmp/gce-startup-script4105198468 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-19876410-1749274874-49-n10cpu16-0001 teamcity-19876410-1749274874-49-n10cpu16-0002 teamcity-19876410-1749274874-49-n10cpu16-0003 teamcity-19876410-1749274874-49-n10cpu16-0004 teamcity-19876410-1749274874-49-n10cpu16-0005 teamcity-19876410-1749274874-49-n10cpu16-0006 teamcity-19876410-1749274874-49-n10cpu16-0007 teamcity-19876410-1749274874-49-n10cpu16-0008 teamcity-19876410-1749274874-49-n10cpu16-0009 teamcity-19876410-1749274874-49-n10cpu16-0010]: failed to run: gcloud compute instances create --subnet default --format json --labels usage=roachtest,arch=arm64,created=2025-06-07t08_11_53z,roachprod=true,cluster=teamcity-19876410-1749274874-49-n10cpu16,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20240319 --image-project ubuntu-os-cloud --boot-disk-type  --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-16 --metadata-from-file startup-script=/tmp/gce-startup-script4105198468 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-19876410-1749274874-49-n10cpu16-0001 teamcity-19876410-1749274874-49-n10cpu16-0002 teamcity-19876410-1749274874-49-n10cpu16-0003 teamcity-19876410-1749274874-49-n10cpu16-0004 teamcity-19876410-1749274874-49-n10cpu16-0005 teamcity-19876410-1749274874-49-n10cpu16-0006 teamcity-19876410-1749274874-49-n10cpu16-0007 teamcity-19876410-1749274874-49-n10cpu16-0008 teamcity-19876410-1749274874-49-n10cpu16-0009 teamcity-19876410-1749274874-49-n10cpu16-0010
stdout: 
stderr: WARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-19876410-1749274874-49-n10cpu16-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-19876410-1749274874-49-n10cpu16-0005].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20240319' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20250606'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
	metric name = compute.googleapis.com/t2a_cpus
	limit name = T2A-CPUS-per-project-region
	limit = 600.0
	dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

cloud=gce
coverageBuild=false
cpu=16
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 07 '25 08:06 cockroach-teamcity

roachtest.cluster_creation failed with artifacts on master @ 44f6c3d374d86abece1e3c19fd0b0bbc7d68b7c7:

test schemachange/random-load failed: (test_runner.go:850).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

cloud=azure
coverageBuild=false
cpu=4
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 07 '25 09:06 cockroach-teamcity

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation failed with artifacts on master @ 44f6c3d374d86abece1e3c19fd0b0bbc7d68b7c7:

test unoptimized-query-oracle/disable-rules=all/seed-multi-region failed: (test_runner.go:850).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

cloud=azure
coverageBuild=false
cpu=4
runtimeAssertionsBuild=true
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 07 '25 10:06 cockroach-teamcity

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation failed with artifacts on master @ 44f6c3d374d86abece1e3c19fd0b0bbc7d68b7c7:

test unoptimized-query-oracle/disable-rules=all/rand-multi-region failed: (test_runner.go:850).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

cloud=azure
coverageBuild=false
cpu=4
runtimeAssertionsBuild=true
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 07 '25 10:06 cockroach-teamcity

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation failed with artifacts on master @ 44f6c3d374d86abece1e3c19fd0b0bbc7d68b7c7:

test unoptimized-query-oracle/disable-rules=half/rand-multi-region failed: (test_runner.go:850).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

cloud=azure
coverageBuild=false
cpu=4
runtimeAssertionsBuild=true
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 07 '25 10:06 cockroach-teamcity

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation failed with artifacts on master @ 44f6c3d374d86abece1e3c19fd0b0bbc7d68b7c7:

test unoptimized-query-oracle/disable-rules=half/seed-multi-region failed: (test_runner.go:850).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

cloud=azure
coverageBuild=false
cpu=4
runtimeAssertionsBuild=true
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 07 '25 10:06 cockroach-teamcity

roachtest.cluster_creation failed with artifacts on master @ dd2376a7fb621089dc0e09a7be9865f792ef5e10:

test kv0/enc=false/nodes=3/cpu=32/mt-shared-process failed: (test_runner.go:850).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-FI0b5lyDaDVhatgSwYlPIai2ASM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19879507-1749361274-22-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-06-08T06:03:44Z},{Key=Name,Value=teamcity-19879507-1749361274-22-n4cpu32-0001},{Key=Cluster,Value=teamcity-19879507-1749361274-22-n4cpu32},{Key=spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19879507-1749361274-22-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-06-08T06:03:44Z},{Key=Name,Value=teamcity-19879507-1749361274-22-n4cpu32-0001},{Key=Cluster,Value=teamcity-19879507-1749361274-22-n4cpu32},{Key=spot,Value=true}] --user-data file:///tmp/aws-startup-script492077470 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping786050115 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

cloud=aws
coverageBuild=false
cpu=32
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 08 '25 06:06 cockroach-teamcity

roachtest.cluster_creation failed with artifacts on master @ dd2376a7fb621089dc0e09a7be9865f792ef5e10:

test kv0/enc=false/nodes=3/cpu=32/size=64kb failed: (test_runner.go:850).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-FI0b5lyDaDVhatgSwYlPIai2ASM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19879507-1749361274-25-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-06-08T06:05:20Z},{Key=Name,Value=teamcity-19879507-1749361274-25-n4cpu32-0002},{Key=Cluster,Value=teamcity-19879507-1749361274-25-n4cpu32},{Key=spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19879507-1749361274-25-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-06-08T06:05:20Z},{Key=Name,Value=teamcity-19879507-1749361274-25-n4cpu32-0002},{Key=Cluster,Value=teamcity-19879507-1749361274-25-n4cpu32},{Key=spot,Value=true}] --user-data file:///tmp/aws-startup-script2379229626 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2618657102 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

cloud=aws
coverageBuild=false
cpu=32
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 08 '25 06:06 cockroach-teamcity

roachtest.cluster_creation failed with artifacts on master @ dd2376a7fb621089dc0e09a7be9865f792ef5e10:

test kv95/enc=false/nodes=3/cpu=32/mvcc-range-keys=global failed: (test_runner.go:850).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-FI0b5lyDaDVhatgSwYlPIai2ASM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=cluster,Value=teamcity-19879507-1749361274-38-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-06-08T06:19:01Z},{Key=Name,Value=teamcity-19879507-1749361274-38-n4cpu32-0002},{Key=Cluster,Value=teamcity-19879507-1749361274-38-n4cpu32},{Key=spot,Value=true},{Key=roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=cluster,Value=teamcity-19879507-1749361274-38-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-06-08T06:19:01Z},{Key=Name,Value=teamcity-19879507-1749361274-38-n4cpu32-0002},{Key=Cluster,Value=teamcity-19879507-1749361274-38-n4cpu32},{Key=spot,Value=true},{Key=roachprod,Value=true}] --user-data file:///tmp/aws-startup-script1800977746 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3360398280 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c7gd.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c7gd.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

cloud=aws
coverageBuild=false
cpu=32
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 08 '25 06:06 cockroach-teamcity

roachtest.cluster_creation failed with artifacts on master @ dd2376a7fb621089dc0e09a7be9865f792ef5e10:

test tpcc-nowait/literal/w=1000/nodes=5/cpu=16 failed: (test_runner.go:850).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.4xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-FI0b5lyDaDVhatgSwYlPIai2ASM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=created,Value=2025-06-08T06:41:00Z},{Key=Name,Value=teamcity-19879507-1749361274-47-n6cpu16-0002},{Key=Cluster,Value=teamcity-19879507-1749361274-47-n6cpu16},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19879507-1749361274-47-n6cpu16},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=created,Value=2025-06-08T06:41:00Z},{Key=Name,Value=teamcity-19879507-1749361274-47-n6cpu16-0002},{Key=Cluster,Value=teamcity-19879507-1749361274-47-n6cpu16},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19879507-1749361274-47-n6cpu16},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64}] --user-data file:///tmp/aws-startup-script112012464 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3763621491 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.4xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.4xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

cloud=aws
coverageBuild=false
cpu=16
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 08 '25 06:06 cockroach-teamcity

roachtest.cluster_creation failed with artifacts on master @ dd2376a7fb621089dc0e09a7be9865f792ef5e10:

test tpccbench/nodes=3/cpu=16 failed: (test_runner.go:850).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.4xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-FI0b5lyDaDVhatgSwYlPIai2ASM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19879507-1749361274-51-n4cpu16},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-06-08T06:43:54Z},{Key=Name,Value=teamcity-19879507-1749361274-51-n4cpu16-0003},{Key=Cluster,Value=teamcity-19879507-1749361274-51-n4cpu16},{Key=spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19879507-1749361274-51-n4cpu16},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-06-08T06:43:54Z},{Key=Name,Value=teamcity-19879507-1749361274-51-n4cpu16-0003},{Key=Cluster,Value=teamcity-19879507-1749361274-51-n4cpu16},{Key=spot,Value=true}] --user-data file:///tmp/aws-startup-script1844143683 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2470888579 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

cloud=aws
coverageBuild=false
cpu=16
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 08 '25 06:06 cockroach-teamcity

roachtest.cluster_creation failed with artifacts on master @ dd2376a7fb621089dc0e09a7be9865f792ef5e10:

test ycsb/C/nodes=3/cpu=32/isolation-level=read-committed failed: (test_runner.go:850).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-FI0b5lyDaDVhatgSwYlPIai2ASM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19879507-1749361274-61-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-06-08T07:02:43Z},{Key=Name,Value=teamcity-19879507-1749361274-61-n4cpu32-0002},{Key=Cluster,Value=teamcity-19879507-1749361274-61-n4cpu32},{Key=spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19879507-1749361274-61-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-06-08T07:02:43Z},{Key=Name,Value=teamcity-19879507-1749361274-61-n4cpu32-0002},{Key=Cluster,Value=teamcity-19879507-1749361274-61-n4cpu32},{Key=spot,Value=true}] --user-data file:///tmp/aws-startup-script2889680445 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3165439686 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

cloud=aws
coverageBuild=false
cpu=32
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 08 '25 07:06 cockroach-teamcity

roachtest.cluster_creation failed with artifacts on master @ dd2376a7fb621089dc0e09a7be9865f792ef5e10:

test ycsb/C/nodes=3/cpu=32/mvcc-range-keys=global failed: (test_runner.go:850).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-FI0b5lyDaDVhatgSwYlPIai2ASM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=created,Value=2025-06-08T07:04:20Z},{Key=Name,Value=teamcity-19879507-1749361274-64-n4cpu32-0001},{Key=Cluster,Value=teamcity-19879507-1749361274-64-n4cpu32},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19879507-1749361274-64-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=created,Value=2025-06-08T07:04:20Z},{Key=Name,Value=teamcity-19879507-1749361274-64-n4cpu32-0001},{Key=Cluster,Value=teamcity-19879507-1749361274-64-n4cpu32},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19879507-1749361274-64-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64}] --user-data file:///tmp/aws-startup-script3452658112 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1542391329 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c7gd.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c7gd.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

cloud=aws
coverageBuild=false
cpu=32
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 08 '25 07:06 cockroach-teamcity

roachtest.cluster_creation failed with artifacts on master @ dd2376a7fb621089dc0e09a7be9865f792ef5e10:

test ycsb/D/nodes=3/cpu=32/isolation-level=read-committed failed: (test_runner.go:850).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-FI0b5lyDaDVhatgSwYlPIai2ASM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=created,Value=2025-06-08T07:05:54Z},{Key=Name,Value=teamcity-19879507-1749361274-67-n4cpu32-0002},{Key=Cluster,Value=teamcity-19879507-1749361274-67-n4cpu32},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19879507-1749361274-67-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=created,Value=2025-06-08T07:05:54Z},{Key=Name,Value=teamcity-19879507-1749361274-67-n4cpu32-0002},{Key=Cluster,Value=teamcity-19879507-1749361274-67-n4cpu32},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19879507-1749361274-67-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64}] --user-data file:///tmp/aws-startup-script1504835385 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1354911342 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

cloud=aws
coverageBuild=false
cpu=32
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 08 '25 07:06 cockroach-teamcity

roachtest.cluster_creation failed with artifacts on master @ dd2376a7fb621089dc0e09a7be9865f792ef5e10:

test ycsb/D/nodes=3/cpu=32/mvcc-range-keys=global failed: (test_runner.go:850).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-FI0b5lyDaDVhatgSwYlPIai2ASM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19879507-1749361274-70-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-06-08T07:07:29Z},{Key=Name,Value=teamcity-19879507-1749361274-70-n4cpu32-0002},{Key=Cluster,Value=teamcity-19879507-1749361274-70-n4cpu32}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19879507-1749361274-70-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-06-08T07:07:29Z},{Key=Name,Value=teamcity-19879507-1749361274-70-n4cpu32-0002},{Key=Cluster,Value=teamcity-19879507-1749361274-70-n4cpu32}] --user-data file:///tmp/aws-startup-script1385789093 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2221823726 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

cloud=aws
coverageBuild=false
cpu=32
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 08 '25 07:06 cockroach-teamcity

roachtest.cluster_creation failed with artifacts on master @ dd2376a7fb621089dc0e09a7be9865f792ef5e10:

test ycsb/E/nodes=3/cpu=32 failed: (test_runner.go:850).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-FI0b5lyDaDVhatgSwYlPIai2ASM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-19879507-1749361274-73-n4cpu32-0001},{Key=Cluster,Value=teamcity-19879507-1749361274-73-n4cpu32},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19879507-1749361274-73-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-06-08T07:09:07Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-19879507-1749361274-73-n4cpu32-0001},{Key=Cluster,Value=teamcity-19879507-1749361274-73-n4cpu32},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-19879507-1749361274-73-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2025-06-08T07:09:07Z}] --user-data file:///tmp/aws-startup-script2881501492 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3808923484 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

cloud=aws
coverageBuild=false
cpu=32
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 08 '25 07:06 cockroach-teamcity

roachtest.cluster_creation failed with artifacts on master @ dd2376a7fb621089dc0e09a7be9865f792ef5e10:

test costfuzz/rand-multi-region failed: (test_runner.go:850).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

cloud=azure
coverageBuild=false
cpu=4
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 08 '25 07:06 cockroach-teamcity

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation failed with artifacts on master @ dd2376a7fb621089dc0e09a7be9865f792ef5e10:

test costfuzz/seed-multi-region failed: (test_runner.go:850).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

cloud=azure
coverageBuild=false
cpu=4
runtimeAssertionsBuild=true
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 08 '25 07:06 cockroach-teamcity

roachtest.cluster_creation failed with artifacts on master @ dd2376a7fb621089dc0e09a7be9865f792ef5e10:

test schemachange/random-load failed: (test_runner.go:850).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

cloud=azure
coverageBuild=false
cpu=4
runtimeAssertionsBuild=false
ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

#147288 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-25.2]
#147281 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147280 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147277 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#147276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
#147274 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
#146983 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.1-rc]
#146939 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
#146719 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.19-rc]

_{This test on roachdash | Improve this report!}

Jun 08 '25 09:06 cockroach-teamcity

cockroach cockroach copied to clipboard

roachtest: cluster_creation failed

cockroach
cockroach copied to clipboard