terraform-provider-heroku icon indicating copy to clipboard operation
terraform-provider-heroku copied to clipboard

Published 20 hours ago • verified publisher icon heroku

Creating multiple apps simultaneously 503

Open talbright opened this issue 6 years ago • 3 comments

Terraform Version

terraform -v
Terraform v0.11.10
+ provider.heroku (unversioned)
+ provider.random v2.0.0

Heroku Provider Version

master (https://github.com/terraform-providers/terraform-provider-heroku/commit/08f5dbac2b1b84ba85fe07fdeed9974789377a1d)

Affected Resource(s)

Please list the resources as a list, for example:

  • heroku_app

Terraform Configuration Files

resource "heroku_space" "default" {
  name         = "test-terraform-provider-heroku"
  organization = "${var.organization}"
  region       = "${var.region}"
}

resource "random_pet" "default" {}

resource "heroku_app" "default" {
  count  = 10
  name   = "${format("test-app-%d-%s",count.index,random_pet.default.id)}"
  space  = "${heroku_space.default.name}"
  region = "${heroku_space.default.region}"

  organization = {
    name = "${heroku_space.default.organization}"
  }
}

Debug Output

heroku_app.default.8: Still creating... (10s elapsed)
heroku_app.default.9: Still creating... (10s elapsed)
heroku_app.default.1: Still creating... (10s elapsed)
heroku_app.default.2: Still creating... (10s elapsed)
heroku_app.default.7: Still creating... (10s elapsed)
heroku_app.default.0: Still creating... (10s elapsed)
heroku_app.default.5: Still creating... (10s elapsed)
heroku_app.default.3: Still creating... (10s elapsed)
heroku_app.default.4: Still creating... (10s elapsed)
heroku_app.default.6: Still creating... (10s elapsed)
heroku_app.default[7]: Creation complete after 11s (ID: test-app-7-measured-mastodon)
heroku_app.default[1]: Creation complete after 15s (ID: test-app-1-measured-mastodon)
heroku_app.default[9]: Creation complete after 19s (ID: test-app-9-measured-mastodon)
heroku_app.default.5: Still creating... (20s elapsed)
heroku_app.default.2: Still creating... (20s elapsed)
heroku_app.default.0: Still creating... (20s elapsed)
heroku_app.default.8: Still creating... (20s elapsed)
heroku_app.default.6: Still creating... (20s elapsed)
heroku_app.default.3: Still creating... (20s elapsed)
heroku_app.default.4: Still creating... (20s elapsed)
heroku_app.default[8]: Creation complete after 23s (ID: test-app-8-measured-mastodon)
heroku_app.default[0]: Creation complete after 25s (ID: test-app-0-measured-mastodon)
heroku_app.default[4]: Creation complete after 28s (ID: test-app-4-measured-mastodon)
heroku_app.default[6]: Creation complete after 29s (ID: test-app-6-measured-mastodon)
heroku_app.default.5: Still creating... (30s elapsed)
heroku_app.default.2: Still creating... (30s elapsed)
heroku_app.default.3: Still creating... (30s elapsed)
heroku_app.default[3]: Creation complete after 32s (ID: test-app-3-measured-mastodon)
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku: 2018/12/23 01:25:22 HTTP/2.0 503 Service Unavailable
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku: Content-Length: 561
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku: Cache-Control: no-cache, no-store
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku: Content-Type: text/html; charset=utf-8
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku: Date: Sun, 23 Dec 2018 06:25:22 GMT
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku: Via: 2.0 spaces-router (d316c7e8a021)
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku: 
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku: <!DOCTYPE html>
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku: <html>
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:   <head>
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:     <meta name="viewport" content="width=device-width, initial-scale=1">
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:     <meta charset="utf-8">
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:     <title>Application Error</title>
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:     <style media="screen">
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:       html,body,iframe {
2018/12/23 01:25:22 [TRACE] root: eval: *terraform.EvalWriteState
2018/12/23 01:25:22 [TRACE] root: eval: *terraform.EvalApplyProvisioners
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:         margin: 0;
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:         padding: 0;
2018/12/23 01:25:22 [TRACE] root: eval: *terraform.EvalIf
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:       html,body {
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:         height: 100%;
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:         overflow: hidden;
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:       }
2018/12/23 01:25:22 [ERROR] root: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:

* heroku_app.default.9: Post https://api.heroku.com/teams/apps: encountered an error : 503 Service Unavailable
2018/12/23 01:25:22 [ERROR] root: eval: *terraform.EvalSequence, err: 1 error(s) occurred:

* heroku_app.default.9: Post https://api.heroku.com/teams/apps: encountered an error : 503 Service Unavailable
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku: 
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:       iframe {
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:         width: 100%;
2018-12-23T01:25:22.777-0500 [DEBUG] plugin.terraform-provider-heroku:         height: 100%;
2018-12-23T01:25:22.777-0500 [DEBUG] plugin.terraform-provider-heroku:         border: 0;
2018-12-23T01:25:22.777-0500 [DEBUG] plugin.terraform-provider-heroku:       }
2018-12-23T01:25:22.777-0500 [DEBUG] plugin.terraform-provider-heroku:     </style>
2018-12-23T01:25:22.777-0500 [DEBUG] plugin.terraform-provider-heroku:   </head>
2018-12-23T01:25:22.777-0500 [DEBUG] plugin.terraform-provider-heroku:   <body>
2018-12-23T01:25:22.777-0500 [DEBUG] plugin.terraform-provider-heroku:     <iframe src="//www.herokucdn.com/error-pages/application-error.html"></iframe>
2018-12-23T01:25:22.777-0500 [DEBUG] plugin.terraform-provider-heroku:   </body>
2018-12-23T01:25:22.777-0500 [DEBUG] plugin.terraform-provider-heroku: </html>

Error: Error applying plan:

2 error(s) occurred:

* heroku_app.default[5]: 1 error(s) occurred:

* heroku_app.default.5: Post https://api.heroku.com/teams/apps: encountered an error : 503 Service Unavailable
* heroku_app.default[2]: 1 error(s) occurred:

* heroku_app.default.2: Post https://api.heroku.com/teams/apps: encountered an error : 503 Service Unavailable

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error

Expected Behavior

All ten apps should have been created.

Actual Behavior

Some were created, then received a 503 error from the platform API.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

  1. terraform apply

talbright avatar Dec 17 '18 00:12 talbright

@mars should we add 503 as a condition to https://github.com/heroku/heroku-go/commit/3b96879aa0a537cdfabc258cd8a13bc079d06801? I suspect the error handling may be off in the back-end, for it to respond with HTML instead of JSON.

talbright avatar Dec 23 '18 06:12 talbright

Hi @talbright 🎄😄

I'd be curious if this is still reproducible, as it appears to be evidence of an incident with the platform API.

I agree that adding a special 503 response status case to the client would be beneficial, so that the provider can sanely handle this "API is offline" condition.

Although I am not familiar with the specific error reported here (creating many apps at once), this kind of Heroku error normally comes from the load balancer when a Heroku app is offline.

mars avatar Dec 23 '18 14:12 mars

🎅 yo!

I did reproduce it last night -- the HCL above has recreated this 💯of the time so far.

talbright avatar Dec 24 '18 00:12 talbright