GCS Destination : Overwrite does not work properly using variables in GCS Bucket Path

[kev-datams]

Connector Name

destination-google-cloud-storage

Connector Version

0.4.4

What step the error happened?

During the sync

Revelant information

Hello,

The GCS destination does not properly apply the Overwrite strategy in certain circumstances. I mean the existing files are not removed from GCS path before the sync, leading to duplicates files after the sync.

In GCS destination configuration settings, while filling GCS Bucket Path value:

• with a simple path like my_folder/my_subfolder : Overwrite strategy works as expected
• with a dynamic path like my_folder/${STREAM_NAME}: Overwrite strategy does not work as expected as explained above.

Reproducibility is really simple:

1. set up a GCS destination with simple path in GCS Bucket Path value
2. create a connection using this GCS destination (source does not matter) using Overwrite strategy
3. sync the connection => a file will be generated as expected
4. re-sync the connection => the file will be dropped first, then re-generated as expected
5. update the GCS destination GCS Bucket Path value with a dynamic path
6. re-sync the connection => the file will NOT be dropped first, then re-generated, leading to duplicates

Thank you for your help 👍

Relevant log output

No response

Contribute

• [ ] Yes, I want to contribute

[kev-datams]

Can someone add the label connectors/destination/gcs to this issue please ?

[kev-datams]

Hi, looks like this issue is still open