terraform-provider-airbyte
terraform-provider-airbyte copied to clipboard
AWS S3 source missing format argument when creating connection resource
Provider version: 0.3.1
Airbyte OS version: 0.50.15
When attempting to create a connection using the terraform provider, I'm running into the following exception.
2023-08-12 22:09:11 │ Error: unexpected response from API. Got an unexpected response code 400
2023-08-12 22:09:11 │
2023-08-12 22:09:11 │ with module.airbyte.airbyte_connection.my_connection,
2023-08-12 22:09:11 │ on airbyte/connectors.tf line 43, in resource "airbyte_connection" "my_connection":
2023-08-12 22:09:11 │ 43: resource "airbyte_connection" "my_connection" {
...
2023-08-12 22:09:11 │ {"type":"https://reference.airbyte.com/reference/errors","title":"unexpected-problem","status":400,"detail":"Something
2023-08-12 22:09:11 │ went wrong in the connector. logs:__init__() missing 1 required positional
2023-08-12 22:09:11 │ argument: 'format'"}
Below is the configuration for my S3 resource:
resource "airbyte_source_s3" "my_source_s3" {
configuration = {
dataset = "Demo"
format = {
source_s3_file_format_jsonl = {
filetype = "jsonl"
newlines_in_values = false
}
}
path_pattern = "*.json"
provider = {
aws_access_key_id = var.aws_user_access_key_id
aws_secret_access_key = var.aws_user_access_key
bucket = var.s3_bucket
endpoint = var.s3_endpoint
}
source_type = "s3"
}
name = "Demo"
workspace_id = airbyte_workspace.my_workspace.workspace_id
}
The AWS S3 source seems to be creating successfully (visible in the UI). However, the format
property in terraform is not being respected.
This only raises an exception when attempting to create the following connector:
resource "airbyte_connection" "my_connection" {
destination_id = airbyte_destination_postgres.my_destination_postgres.destination_id
name = "${airbyte_source_s3.my_source_s3.name} -> ${airbyte_destination_postgres.my_destination_postgres.name}"
schedule = {
schedule_type = "manual"
}
source_id = airbyte_source_s3.my_source_s3.source_id
}