DataflowTemplates icon indicating copy to clipboard operation
DataflowTemplates copied to clipboard

[Bugfix] Replace only the last occurrence of Datetime patterns in a directory path

Open ahmedabu98 opened this issue 1 year ago • 3 comments

While using a template that utilizes DLQ, found some cases where dead-letter GCS directory names get altered unexpectedly if they contain Datetime patterns.

for example: DLQ would take gs://bucket/dead-letter/recommendation and add on to it /YYYY/MM/DD/HH/mm then resolve for datetime patterns. The resulting path would look like this: gs://bucket/dead-letter/reco56endation/2023/06/15/14/56. All occurrences of patterns get replaced. The changes in this PR make a more accurate best effort by only replacing the last occurrence in the path.

ahmedabu98 avatar Jun 15 '23 18:06 ahmedabu98

R: @bvolpato

ahmedabu98 avatar Jun 15 '23 18:06 ahmedabu98

This is a risky change - are we sure that there are no customers relying on the old behavior?

reuvenlax avatar Jun 17 '23 05:06 reuvenlax

@reuvenlax agreed. pranav and I decided to just expose a parameter to configure the datetime pattern

ahmedabu98 avatar Jun 21 '23 18:06 ahmedabu98

This pull request has been marked as stale due to 180 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time. Thank you for your contributions.

github-actions[bot] avatar May 20 '24 14:05 github-actions[bot]

This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

github-actions[bot] avatar May 28 '24 02:05 github-actions[bot]