[Roadmap] Enable Field/Column Selection through the API

[nataliekwong]

As requested here, enable column or field selection to be available or configurable through the application / UI. This is not currently available through the API.

Definition of done:

• Ability to view the catalog (field names and types) for all discovered streams
• Ability to enable or disable fields within each stream and save configuration

This will be available for all Airbyte products.

[AlexDouglasUK]

Any update on this? We would love this option

[nataliekwong]

Hey @AlexDouglasUK , thanks for your comment! We're currently planning for the next quarter. This has been raised as a priority to our Engineering, but not yet committed as we've been focused on making our syncs more reliable the first half of this year. If this does get committed, we'll add it here: https://go.airbyte.com/roadmap.

[nataliekwong]

New target release date of July 2024.

[nataliekwong]

This has now been released as a part of v0.63.4.

See the commit: https://github.com/airbytehq/airbyte-platform/commit/34d13bd1b09daf98ed74972cb9e4c676cd9806b3, as the API docs have yet to be updated.

[sowla]

Thank you for the functionality and keeping us updated @nataliekwong! Would it be possible to share if adding this functionality to the terraform provider (https://github.com/airbytehq/terraform-provider-airbyte/issues/69) is also planned..?

[dantonbertuol]

This has now been released as a part of v0.63.4.

See the commit: airbytehq/airbyte-platform@34d13bd, as the API docs have yet to be updated.

Wooow this is amazing.

[nataliekwong]

We have not yet planned on when it will be added to the terraform provider. While this issue is closed, I will update here when it has been added.

[pankaj-lmg]

Seems like it is added to the terraform provider, but not working. Ref: https://registry.terraform.io/providers/airbytehq/airbyte/latest/docs/data-sources/connection#selected_fields

[pankaj-lmg]

Seems like fieldSelectionEnabled is not exposed via terraform-provider, and hence it is always going as false. Due to which below happens in platform:

@nataliekwong possible to get it fixed?

[nataliekwong]

Hi there, we have not yet planned when we will make this available in the Terraform Provider. I don't have an ETA at this time, but you can upvote or comment on this issue.

[Iyanumanuel]

Okay, thank you @nataliekwong for the updates. Please let us know when it is available.

[dantonbertuol]

Hi @nataliekwong, i need an example using the selectedFields property. It's not in the documentation yet.

[dantonbertuol]

Analyzing the source code I managed to develop an example, maybe it will help other people:

"configurations":
        {
            "streams": [
                {"name": "table_name", "syncMode": "incremental_append", "selectedFields": [{"fieldPath": "field_name"}]}
             ]
        }

[kev-datams]

@dantonbertuol will you contribute to propose a PR ? 🙏