redpanda icon indicating copy to clipboard operation
redpanda copied to clipboard

CI Failure (`Quota 'NETWORKS' exceeded`) in `SelfRedpandaCloudTest.test_simple`

Open vbotbuildovich opened this issue 1 year ago • 11 comments

https://buildkite.com/redpanda/vtools/builds/11776 https://buildkite.com/redpanda/vtools/builds/11796 https://buildkite.com/redpanda/vtools/builds/11821 https://buildkite.com/redpanda/vtools/builds/11837

Module: rptest.redpanda_cloud_tests.cloud_self_test
Class: SelfRedpandaCloudTest
Method: test_simple
test_id:    SelfRedpandaCloudTest.test_simple
status:     FAIL
run time:   44.170 seconds

RpkException('command /opt/redpanda/bin/rpk cloud byoc gcp apply --redpanda-id=cn2o866m2v5uhk75ck5g --project-id=devprod-cicd-infra -v returned 1, output: ', '2024-02-09T01:47:16.437Z\t\x1b[34mINFO\x1b[0m\t.rpk.managed-byoc\tvalidation/validate.go:68\tChecking project... PASSED\n2024-02-09T01:47:17.265Z\t\x1b[34mINFO\x1b[0m\t.rpk.managed-byoc\tgcp/apply.go:76\tReconciling agent infrastructure...\n2024-02-09T01:47:17.492Z\t\x1b[34mINFO\x1b[0m\t.rpk.managed-byoc\tcli/cli.go:190\tRunning apply\t{"provisioner": "bootstrap"}\n2024-02-09T01:47:23.291Z\t\x1b[34mINFO\x1b[0m\t.rpk.managed-byoc\tcli/cli.go:200\tFinished apply\t{"provisioner": "bootstrap"}\n2024-02-09T01:47:23.291Z\t\x1b[34mINFO\x1b[0m\t.rpk.managed-byoc\tcli/cli.go:190\tRunning apply\t{"provisioner": "network"}\n2024-02-09T01:47:42.533Z\t\x1b[31mERROR\x1b[0m\t.rpk.managed-byoc\tcli/cli.go:193\tFailed to apply provisioner\t{"provisioner": "network", "error": "failed running terraform apply: exit status 1\\n\\nError: Error waiting to create Network: Error waiting for Creating Network: Quota \'NETWORKS\' exceeded.  Limit: 30.0 globally.\\n\\tmetric name = compute.googleapis.com/networks\\n\\tlimit name = NETWORKS-per-project\\n\\tlimit = 30\\n\\tdimensions = map[global:global]\\n\\n\\n  with google_compute_network.redpanda[0],\\n  on main.tf line 53, in resource \\"google_compute_network\\" \\"redpanda\\":\\n  53: resource \\"google_compute_network\\" \\"redpanda\\" {\\n\\n"}\nFailed to apply provisioners: failed to apply provisioner network: failed running terraform apply: exit status 1\n\nError: Error waiting to create Network: Error waiting for Creating Network: Quota \'NETWORKS\' exceeded.  Limit: 30.0 globally.\n\tmetric name = compute.googleapis.com/networks\n\tlimit name = NETWORKS-per-project\n\tlimit = 30\n\tdimensions = map[global:global]\n\n\n  with google_compute_network.redpanda[0],\n  on main.tf line 53, in resource "google_compute_network" "redpanda":\n  53: resource "google_compute_network" "redpanda" {\n\n\n', 1, '')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 182, in _do_run
    self.setup_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 260, in setup_test
    self.test.setup()
  File "/home/ubuntu/redpanda/tests/rptest/tests/redpanda_cloud_test.py", line 40, in setup
    super().setup()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/test.py", line 91, in setup
    self.setUp()
  File "/home/ubuntu/redpanda/tests/rptest/tests/redpanda_test.py", line 38, in setUp
    self.__redpanda.start()
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1638, in start
    cluster_id = self._cloud_cluster.create(superuser=superuser)
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda_cloud.py", line 901, in create
    self._create_new_cluster()
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda_cloud.py", line 686, in _create_new_cluster
    self.utils.rpk_cloud_apply(_cluster_id)
  File "/home/ubuntu/redpanda/tests/rptest/services/cloud_cluster_utils.py", line 132, in rpk_cloud_apply
    out = self._exec(cmd, timeout=1800)
  File "/home/ubuntu/redpanda/tests/rptest/services/cloud_cluster_utils.py", line 103, in _exec
    return self.rpk._execute(cmd, env=self.env, timeout=timeout)
  File "/home/ubuntu/redpanda/tests/rptest/clients/rpk.py", line 1082, in _execute
    raise RpkException(
rptest.clients.rpk.RpkException: RpkException<command /opt/redpanda/bin/rpk cloud byoc gcp apply --redpanda-id=cn2o866m2v5uhk75ck5g --project-id=devprod-cicd-infra -v returned 1, output: ; stderr: 2024-02-09T01:47:16.437Z	^[[34mINFO^[[0m	.rpk.managed-byoc	validation/validate.go:68	Checking project... PASSED
2024-02-09T01:47:17.265Z	^[[34mINFO^[[0m	.rpk.managed-byoc	gcp/apply.go:76	Reconciling agent infrastructure...
2024-02-09T01:47:17.492Z	^[[34mINFO^[[0m	.rpk.managed-byoc	cli/cli.go:190	Running apply	{"provisioner": "bootstrap"}
2024-02-09T01:47:23.291Z	^[[34mINFO^[[0m	.rpk.managed-byoc	cli/cli.go:200	Finished apply	{"provisioner": "bootstrap"}
2024-02-09T01:47:23.291Z	^[[34mINFO^[[0m	.rpk.managed-byoc	cli/cli.go:190	Running apply	{"provisioner": "network"}
2024-02-09T01:47:42.533Z	^[[31mERROR^[[0m	.rpk.managed-byoc	cli/cli.go:193	Failed to apply provisioner	{"provisioner": "network", "error": "failed running terraform apply: exit status 1\n\nError: Error waiting to create Network: Error waiting for Creating Network: Quota 'NETWORKS' exceeded.  Limit: 30.0 globally.\n\tmetric name = compute.googleapis.com/networks\n\tlimit name = NETWORKS-per-project\n\tlimit = 30\n\tdimensions = map[global:global]\n\n\n  with google_compute_network.redpanda[0],\n  on main.tf line 53, in resource \"google_compute_network\" \"redpanda\":\n  53: resource \"google_compute_network\" \"redpanda\" {\n\n"}
Failed to apply provisioners: failed to apply provisioner network: failed running terraform apply: exit status 1

Error: Error waiting to create Network: Error waiting for Creating Network: Quota 'NETWORKS' exceeded.  Limit: 30.0 globally.
	metric name = compute.googleapis.com/networks
	limit name = NETWORKS-per-project
	limit = 30
	dimensions = map[global:global]


  with google_compute_network.redpanda[0],
  on main.tf line 53, in resource "google_compute_network" "redpanda":
  53: resource "google_compute_network" "redpanda" {


; returncode: 1>

JIRA Link: CORE-1782

vbotbuildovich avatar Feb 13 '24 20:02 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/11854

vbotbuildovich avatar Feb 15 '24 00:02 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/11878

vbotbuildovich avatar Feb 17 '24 00:02 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/11889

vbotbuildovich avatar Feb 18 '24 00:02 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/11898

vbotbuildovich avatar Feb 19 '24 00:02 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/11908

vbotbuildovich avatar Feb 20 '24 00:02 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/11935

vbotbuildovich avatar Feb 21 '24 00:02 vbotbuildovich

fixed in issue https://github.com/redpanda-data/cloudv2/issues/12825

andrewhsu avatar Feb 21 '24 04:02 andrewhsu

*https://buildkite.com/redpanda/vtools/builds/12453

vbotbuildovich avatar Mar 22 '24 21:03 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/12529 *https://buildkite.com/redpanda/vtools/builds/12533

vbotbuildovich avatar Mar 29 '24 04:03 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/12676 *https://buildkite.com/redpanda/vtools/builds/12677 *https://buildkite.com/redpanda/vtools/builds/12680 *https://buildkite.com/redpanda/vtools/builds/12691 *https://buildkite.com/redpanda/vtools/builds/12710 *https://buildkite.com/redpanda/vtools/builds/12680

vbotbuildovich avatar Apr 04 '24 21:04 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/12730 *https://buildkite.com/redpanda/vtools/builds/12731 *https://buildkite.com/redpanda/vtools/builds/12735 *https://buildkite.com/redpanda/vtools/builds/12734

vbotbuildovich avatar Apr 10 '24 17:04 vbotbuildovich

Duplicate of https://github.com/redpanda-data/redpanda/issues/16587 and already fixed

rpdevmp avatar Jul 26 '24 19:07 rpdevmp