airbytehq
[destination prefix] Cant add "/" to Destination Stream Prefix
Topic
"/" seem to be seen as invalid character in Destination Stream Prefix since version 0.50.3
Relevant information
In version 0.50.51 it looks like this when trying to add a slash in destination stream prefix:
And like this in version 0.50.3:
We've used slashes to extract data into separate directories in object storage (azure blob storage) using the same destination connector.
To achieve the same result now we need to setup multiple destination connector where we specify the directory in the container propery, like "container_name/directory".
It was nice to previolusly have been able to do it in each connection and reusing the destination.
I think this is a design decision because not all databases allow to create tables or objects using _. This configuration today uses a generic rule instead of being specific to each destination. For now the workaround is to remove that for another char container_name_directory.
I think this is a design decision because not all databases allow to create tables or objects using
_. This configuration today uses a generic rule instead of being specific to each destination. For now the workaround is to remove that for another charcontainer_name_directory.
Ok, yes I thought something like that would be the reason. But it's been a really nice way for us to simplify the setup when we just need to setup one Azure blob storage destination and then decide the path/dir in each connection. Instead of, as now, we have to setup multiple destinations for it.
For now the workaround is to remove that for another char
container_name_directory.
You mean that the solution could be to add a new connection property where the user can specify dir? Yes, that'd work great for our use case.
Refinement Notes:
- We need to check here with the destination team what's the supposed way a (file) connectors would handle a
/as. Is the subdirectory creation just happening by accident, or is this a clear documented feature. If so, how can a connector communicate to us (new metadata?) that it'll support the/for the UI to allow it. @nataliekwong to check with team
Require more information from the destination team. The metadata should tell us what character are allowed
From destinations: IMO it seems like something we should be able to support generally. For S3, GCS and ABS - I think this is a non issue (directories aren't real) for the File system ones (Local JSON, Local CSV) I can't think of a good reason we shouldn't support it
At Airbyte, we seek to be clear about the project priorities and roadmap. This issue has not had any activity for 180 days, suggesting that it's not as critical as others. It's possible it has already been fixed. It is being marked as stale and will be closed in 20 days if there is no activity. To keep it open, please comment to let us know why it is important to you and if it is still reproducible on recent versions of Airbyte.
This issue has been fixed because we are no longer validating the stream prefix.
In this PR, we removed SyncCatalog v1 - related code and obsolete components, including the DestinationStreamNamesModal, where the validation that was causing this issue lived.
The stream prefix is now managed in the connection settings, and does not have any validation. This raises the question of whether we should add validation back in; here's the issue that motivated it in the first place. If we do add it back in, we should consider separate validation for each connector subtype.
cc @nataliekwong
