terraform-provider-airbyte icon indicating copy to clipboard operation
terraform-provider-airbyte copied to clipboard

AWS S3 source missing format argument when creating connection resource

Open mitchealex1 opened this issue 1 year ago • 0 comments

Provider version: 0.3.1 Airbyte OS version: 0.50.15

When attempting to create a connection using the terraform provider, I'm running into the following exception.

2023-08-12 22:09:11 │ Error: unexpected response from API. Got an unexpected response code 400
2023-08-12 22:09:11 │ 
2023-08-12 22:09:11 │   with module.airbyte.airbyte_connection.my_connection,
2023-08-12 22:09:11 │   on airbyte/connectors.tf line 43, in resource "airbyte_connection" "my_connection":
2023-08-12 22:09:11 │   43: resource "airbyte_connection" "my_connection" {
...
2023-08-12 22:09:11 │ {"type":"https://reference.airbyte.com/reference/errors","title":"unexpected-problem","status":400,"detail":"Something
2023-08-12 22:09:11 │ went wrong in the connector. logs:__init__() missing 1 required positional
2023-08-12 22:09:11 │ argument: 'format'"}

Below is the configuration for my S3 resource:

resource "airbyte_source_s3" "my_source_s3" {
  configuration = {
    dataset = "Demo"
    format = {
      source_s3_file_format_jsonl = {
        filetype           = "jsonl"
        newlines_in_values = false
      }
    }
    path_pattern = "*.json"
    provider = {
      aws_access_key_id     = var.aws_user_access_key_id
      aws_secret_access_key = var.aws_user_access_key
      bucket                = var.s3_bucket
      endpoint              = var.s3_endpoint
    }
    source_type = "s3"
  }
  name         = "Demo"
  workspace_id = airbyte_workspace.my_workspace.workspace_id
}

The AWS S3 source seems to be creating successfully (visible in the UI). However, the format property in terraform is not being respected. Screen Shot 2023-08-12 at 10 25 24 PM

This only raises an exception when attempting to create the following connector:

resource "airbyte_connection" "my_connection" {
  destination_id = airbyte_destination_postgres.my_destination_postgres.destination_id
  name           = "${airbyte_source_s3.my_source_s3.name} -> ${airbyte_destination_postgres.my_destination_postgres.name}"
  schedule = {
    schedule_type = "manual"
  }
  source_id = airbyte_source_s3.my_source_s3.source_id
}

mitchealex1 avatar Aug 12 '23 10:08 mitchealex1