Terraform provider bug - unexpected response from API. Got an unexpected response code 400 - Primary key for stream: Account is already pre-defined

[tdebroc]

Dear Airbyte team

unexpected response from API. Got an unexpected response code 40 I created a connection in TF like this:

resource "airbyte_connection" "my_connection" {
  data_residency                       = "eu"
  destination_id                       = airbyte_destination_bigquery.bigquery.destination_id
  name                                 = "Connection SFSC to BQ for Account table"
  namespace_definition                 = "custom_format"
  namespace_format                     = "My_namespace"
  non_breaking_schema_updates_behavior = "propagate_columns"
  source_id                            = airbyte_source_salesforce.source_salesforce_account.source_id
  status                               = "active"
  schedule = {
    schedule_type = "manual"
  }
  configurations                       = {
    streams = [
      {
        "name"         = "Account"
        "sync_mode"    = "full_refresh_append"
      }
    ]
  }
}

But when I want to update it. I got:

{"detail":"The body of the request contains an invalid connection configuration. Primary key for stream: Account is already pre-defined. Please do NOT include a primary
│ key configuration for this stream.","type":"[https://reference.airbyte.com/reference/errors","title":"bad-request","status":400](https://reference.airbyte.com/reference/errors%22,%22title%22:%22bad-request%22,%22status%22:400)}

As you can see, I never provided primaryKey. Airbyte automatically has added it when creating the source. I tried to remove it manually in the TF state, but even that don't work, because it re-fetch all the source config before re-applying. And therefore the PATCH doesn't work,

I guess there is a bug in the Terraform provider.

This prevents us from migrating to Terraform which has impact on us. We want to migrate ASAP to TF otherwise we'll accumulate legacy code.

Thanks

Best regards

[octavia-squidington-iii]

Zendesk ticket #5512 has been linked to this issue.

[JonsSpaghetti]

Can you please try upgrading to Airbyte version https://github.com/airbytehq/airbyte-platform/releases/tag/v0.57.3 and try again?

[tdebroc]

Hey @JonsSpaghetti , looks like it works. Thank you very much for the fix,

Best regards

Thibaut

[GabrielZangCubDigital]

resource "airbyte_connection" "my_connection" {
  configurations = {
    streams = [{
      name         = "Product"
      sync_mode    = "incremental_deduped_history"
      cursor_field = ["llCode"]
    }]
  }
  name           = "Mssql to Postgres"
  source_id      = airbyte_source_mssql.my_source_mssql.source_id
  destination_id = airbyte_destination_postgres.my_destination_postgres.destination_id
}

when i update my sync_mode to "incremental_deduped_history". I still got above issue.

airbyte_connection.my_connection: Modifying... [name=Mssql to Postgres]
╷
│ Error: unexpected response from API. Got an unexpected response code 400
│
│   with airbyte_connection.my_connection,
│   on main.tf line 65, in resource "airbyte_connection" "my_connection":
│   65: resource "airbyte_connection" "my_connection" {
│
│ **Request**:
│ PATCH /v1/connections/a72d6de6-a3e3-423e-9beb-98f9bb9a42cd HTTP/1.1
│ Host: localhost:8006
│ Accept: application/json
│ Authorization: Basic YWlyYnl0ZTpwYXNzd29yZA==
│ Content-Type: application/json
│ User-Agent: speakeasy-sdk/go 0.4.1 2.262.2 1.0.0 airbyte
│
│ {"configurations":{"streams":[{"cursorField":["llCode"],"name":"Product","primaryKey":[["llCode"]],"syncMode":"incremental_deduped_history"}]},"dataResidency":"auto","name":"Mssql to
│ Postgres","namespaceDefinition":"destination","namespaceFormat":null,"nonBreakingSchemaUpdatesBehavior":"ignore","schedule":{"scheduleType":"manual"},"status":"active"}
│ **Response**:
│ HTTP/1.1 400 Bad Request
│ Content-Length: 296
│ Connection: keep-alive
│ Content-Type: application/problem+json
│ Date: Mon, 22 Apr 2024 00:52:06 GMT
│ Server: nginx/1.25.3
│
│ {"detail":"The body of the request contains an invalid connection configuration. Primary key for stream: Product is already pre-defined. Please do NOT include a primary key configuration for this
│ stream.","type":"https://reference.airbyte.com/reference/errors","title":"bad-request","status":400}

[vishalprazn]

Hello All

I am too getting same issue:

│ {"type":"https://reference.airbyte.com/reference/errors","title":"bad-request","status":400,"detail":"The 132 │ body of the request contains an invalid connection configuration. Primary 133 │ key for stream: Account is already pre-defined. Please do NOT include a 134 │ primary key configuration for this stream."}

Difficult to progress, please advise.

Resolution (Dirty): Destroyed and Rebuild the connection.

[GabrielZangCubDigital]

Hello All

I am too getting same issue:

│ {"type":"https://reference.airbyte.com/reference/errors","title":"bad-request","status":400,"detail":"The 132 │ body of the request contains an invalid connection configuration. Primary 133 │ key for stream: Account is already pre-defined. Please do NOT include a 134 │ primary key configuration for this stream."}

Difficult to progress, please advise.

Resolution (Dirty): Destroyed and Rebuild the connection.

  #   streams = [{
  #     name         = "Product"
  #     sync_mode    = "incremental_deduped_history"
  #     cursor_field = ["llCode"]
  #     primaryKey   = []
  #   }]

In another github issue, cannot find related link, i found the temporary solution above which works for me,

add an empty primaryKey attribute, it doesnt delete primarykey

[tdebroc]

Hello,

In fact the bug is not solved indeed :-( We just tried our prod migration this migration from Octavia to TF, but are completely blocked by this bug.

The workarounds can't work:

• Adding an empty primary key will make the plan says that there is changes each time we do a tf apply / tf plan which make the usage of TF impossible to work and see the real changes.
• Destroying the connection is not a solution, as this means it will destroy all data related (cursor...à

This is a huge blocker for our Terraform migration, we hope it can get fix soon,

Thanks much in advance,

[tdebroc]

In fact bug is fixed on the airbyte-server, not airbyte-server-api

So switching to terraform provider to this url:

server_url = "${var.AIRBYTE_URL}/api/public/v1"

fixed it :-) 🎉

[tomascorrea]

Hello All

I'm still geting this error with the airbyte latest version (v0.58.1) and terraform version 0.4.2. Changing the server_url didn't work for me.

Thanks much in advance,

[hongbo-miao]

Thanks @tdebroc !

For future people who come here, I want to point out today is actually just simply switching Terraform provider to server_url = "${var.airbyte_url}/api/public/v1". Here airbyte_url points to airbyte-webapp-svc.airbyte-namespace.

It helps

• Terraform works well with new endpoint.
• This new endpoint is latest. airbyte-api-server is actually out-of-date today. The service underneath this new endpoint fixed bug like this one.
• If you use Helm chart to deploy, you can disable airbyte-api-server which is one less server!

  airbyte-api-server:
    enabled: false
  

• Ingress only needs to expose one endpoint now.