Fail when trying to use postgressql provider against aws rds instance in same definition

[hashibot]

This issue was originally opened by @chaliy as hashicorp/terraform#7442. It was migrated here as part of the provider split. The original body of the issue is below.

---

I am trying to implement PostgreSQL database on AWS RDS. I am crating PostgreSQL RDS Instance, and then database"

provider "aws" {}
resource "aws_db_instance" "main" {}

provider "postgresql" {}
resource "postgresql_database" "prototype" {}

Unfortunately this does not work. Provider postgresql for plan requires connection, and it is not yet here, because instance is not created.

postgresql_database.prototype: Error reading info about database: dial tcp: lookup ${aws_db_instance.address}: getaddrinfow: No such host is known.

or

cannot parse '' as int: strconv.ParseInt: parsing "": invalid syntax

Expected Behavior

It was expected that somehow postgresql provider will figure out that instance is not here, and will not crash.

Actual Behavior

Deployment crashes with error

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. https://gist.github.com/chaliy/6af4c607afb04e356ca240f9df45d066
2. terraform plan

[sean-]

@chaliy , this won't exactly help the above issue, but I'd be curious as to your thoughts after reading the following:

https://github.com/terraform-providers/terraform-provider-postgresql/issues/5#issuecomment-314031307

The problem is that the underlying database connection pooler (sql.DB) doesn't allow for post-connection initialization hooks in the same way that github.com/jackc/pgx does. #11 has been opened to track the migration to jackc/pgx. Specifically, this nugget of functionality is required to solve this issue is:

https://github.com/jackc/pgx/blob/master/conn_pool.go#L12 https://godoc.org/github.com/jackc/pgx#ConnPoolConfig

I want to use jackc/pgx for a few reasons unrelated to your issue, but late-binding connections from the provider is one of them. This will prevent terraform plan from attempting to establish new connections.

[tomelliff]

Is this still true? I thought that might be the issue when I started trying to have Terraform create an RDS instance and then add resources but it seems to plan and apply out okay from scratch for me.

I can provide an example of this working if it helps at all.

[sean-]

@tomelliff Is what still true?

[tomelliff]

That it fails to plan when using the postgres provider in the same Terraform config as the RDS instance and using the aws_db_instance.resource_name.address as the host for the postgresql provider details. Because it seems to work fine for me.

[sean-]

Oh, I think I understand what you're saying. Namely that the PostgreSQL resource fails to plan because it has a dependency on the RDS instance being available, but if you attempt to mix and match the terraform to create the RDS instance with plan to create the schema, the provider fails to plan because it can't connect to the yet-to-be-created instance.

There are two possible outcomes here:

1. Create a provider-level dependency where the postgresql provider has a dependency on the rds provider (this doesn't exist yet).
2. Have the provider return an empty config suggesting everything needs to happen in the plan.

I don't like #2 because it could be a transient connectivity problem that would result in a plan being created that would attempt to recreate the database schema. I think the current behavior, while not optimal, is correct for the time being.

The ideal solution would be for there to be a second-round of planning that would happen after the RDS instance is provisioned. I'm not sure how to do that at plan time, however. I need to think more about this, but that's my current thinking.

[tomelliff]

I was asking if the original poster still had issues because it works fine for me as is. Unfortunately the config they pasted is too minimal to reproduce it but as far as I can see this issue could be closed.

[sean-]

@tomelliff if you can post sample configs, that'd be great!

[gabro]

@tomelliff I'm facing the exact same issue as the OP, so if you could post your config it'd be great!

[tomelliff]

I was wrong apparently, plans before the RDS instance are created fail but you can apply out anyway and that manages the dependency order fine.

[gabro]

So you mean something like

terraform apply -target aws_rds_instance.my_instance
terraform apply

?

[tomelliff]

No, just a straight apply. I run my Terraform through some helper scripts but the apply command is terraform apply -input=false -refresh=true -auto-approve=true. It seems to delay anything to do with the Postgres provider until the database is created thanks to the dependency between them by doing something like:

resource "aws_db_instance" "database" {
  ...
}

provider "postgresql" {
  version = "~>0.1"

  host            = "${aws_db_instance.database.address}"
  port            = "${var.database_port}"
  username        = "${var.database_admin_user}"
  password        = "${aws_ssm_parameter.admin_password.value}"
  sslmode         = "require"
  connect_timeout = 15
}

[tomelliff]

I've been considering requiring plans for every apply (and having CI pass the plan to the apply stage) but not overly sure if it's necessary because I'm not massively risk averse and have reasonable test environments before production. That said, if I did have this requirement then the inability to plan a database before applying it the first time would be a deal breaker. It's probably not an easy thing to fix though.

[gabro]

Ah, thanks for the pointers! I'll take a look

[bloo]

I'm seeing this during plan when running on Atlas (Terraform Enterprise), which runs on Ubuntu, but not locally! - on my Darwin/OS X machine. Are there different versions of the provider plugin? I'll be watching this closely and will gladly provide more info if needed.

(fyi - version 0.11.2)

[sean-]

@bloo Can you paste your error message from your plan?

@tomelliff What do you have "${var.database_port}" set to?

[bloo]

@sean- I keep it as its default, but I just tried it set to 5432 and saw the same issue.

Error: Error refreshing state: 1 error(s) occurred:

* module.rds-pg.provider.postgresql: Error initializing PostgreSQL client: error detecting capabilities: error PostgreSQL version: dial tcp :5432: getsockopt: connection refused

[sean-]

@bloo in your run, can you set the logging level to INFO and re-run? There should be a line that looks like:

[INFO] PostgreSQL DSN: `your DSN here`

that will help shed some light on this.

[bloo]

@sean- here's the output with TF_LOG=INFO set:

https://gist.github.com/bloo/638a769ac134bbff4d43974fadafbca5

I couldn't find PostgreSQL DSN anywhere. Fyi, the rds-pg module works because we applied it with provider postegresql commented out, then applied a 2nd time w/ the databases. rds-pg-test is a combination of rds instance and postgresql provider, which fails as previously observed.

[bloo]

@sean- any headway into this? Is there anything else I can provide?

[radeksimko]

Hi folks, sorry this isn't working as expected.

As it was indirectly mentioned this is related to a known core issue where we perform the interpolation (of the DB hostname in this case) too early in the lifecycle.

See e.g. https://github.com/hashicorp/terraform/issues/12393 which was later consolidated to https://github.com/hashicorp/terraform/issues/4149

In regards to workarounds - some have been mentioned already:

Separate out aws and postgres configs

Keep each in its own directory and apply each separately while leveraging the remote state to pass the data in between.

Example:

/aws/main.tf

terraform {
  backend "s3" {
    bucket = "mybucket"
    key    = "path/to/my/key"
    region = "us-east-1"
  }
}

resource "aws_db_instance" "main" {
...
}

output "host" {
  value = "${aws_db_instance.main.address}"
}
output "port" {
  value = "${aws_db_instance.main.port}"
}
output "username" {
  value = "${aws_db_instance.main.username}"
}
output "password" {
  value = "${aws_db_instance.main.password}"
}

/postgres/main.tf

data "terraform_remote_state" "aws" {
  backend = "s3"

  config {
    bucket = "mybucket"
    key    = "path/to/my/key"
    region = "us-east-1"
  }
}

provider "postgresql" {
  alias = "repro"
  host = "${data.terraform_remote_state.aws.host}"
  port = "${data.terraform_remote_state.aws.port}"
  username = "${data.terraform_remote_state.aws.username}"
  password = "${data.terraform_remote_state.aws.password}"
  sslmode = "require"
}

resource "postgresql_database" "repro1" {
  provider = "postgresql.repro"
  name = "repro1"
}

resource "postgresql_database" "repro2" {
  provider = "postgresql.repro"
  name = "repro2"
}

You can read more about remote state here.

Perform two-staged apply

terraform apply -target=aws_db_instance.main && terraform apply

The first apply will create the database instance, save all the details into state and the second apply will start at the point where the instance is already up with all necessary details (like hostname) stored in the state.

Read more about targeting here.

---

I'd recommend subscribing to the mentioned core issue if you want to hear about updates (i.e. any progress on resolving that issue).

[bloo]

Hi @radeksimko - we had hoped to not split our module up if the fix was going to happen soon(ish), but seeing as the the core issue is 2+ years old, do you recommend that we move forward with the split and consider this issue as the state of the world for now?

[radeksimko]

@bloo fix for the mentioned issue is not scheduled for the upcoming release (0.12) and I can't comment on the roadmap beyond that, so I'd recommend you do that.

Thanks for your understanding.

[bloo]

@radeksimko thanks for the transparency 👍

[tomelliff]

I noticed the other day that while I can't plan a config that has an RDS instance to be created plus any Postrgesql resources as others are trying I can apply it from scratch with the 0.1 provider version but this is broken in 0.1.1 (presumably due to the feature checking in https://github.com/terraform-providers/terraform-provider-postgresql/pull/12/files).

Note that I'm also using Terraform 0.10.8 at the moment but am about to bump that to latest (0.11.4) and I'm unsure if the apply without a plan will work in 0.11 with the forced plan. I'm hoping -auto-approve=true should solve that

[mwz]

Hi, I'm experiencing the same issue, has there been any progress on this?

[tburow]

bump on this issue. still seeing this, im really surprised that this hasnt been resolved?

will this be resolved in the 12 release or? ETA on the 12 release?

[cyrilgdn]

Hi,

As very well explained by @radeksimko, this is a Terraform core issue. I recommend everyone concerned to read the aforementioned issue: https://github.com/hashicorp/terraform/issues/4149.

So I'm in favor to close this issue. Any opinion?

[tomelliff]

The 0.1.0 version of the provider allows a clean apply of creating an RDS instance and database in a single action.

Unfortunately the 0.1.1 release introduced a regression in that behaviour because it attempts to do feature detection early on and falls over.

I've not had the chance to really look into fixing it because pinning to 0.1.0 works for us (although prevents us using the fix I raised to allow us to destroy the same setup that was also released in 0.1.1) but I'd like to see if that behaviour can be shunted behind configuration that allows it to be disabled so this type of functionality works even with the limitations in core Terraform that has been previously mentioned.

[cyrilgdn]

@tomelliff Thanks for the clarification. I just tested this case in 0.1.0 and I get your point. I'll try to check what can be done about that.

[milosgajdos]

Silly question, but if the advice is to split your aws and postgres configs into two separate directories, how does one deal with multliple inits?

First, one needs to init the aws config (which generates local .terraform directory), then the postgres config requires to be intiialized too which if done from the "root" directory it will override the previously created .terraform directory with postgres config one. Is there any terraform magic trick I'm missing?