snowplow-rdb-loader icon indicating copy to clipboard operation
snowplow-rdb-loader copied to clipboard
verified publisher icon snowplow

RDB Shredder: fix folder discovery at the root of the bucket

Open chuwy opened this issue 3 years ago • 0 comments

If user saves the data at the root of the bucket, like:

s3://snowplow-bucket/run=2022-01-28-16-30-00/

Instead of:

s3://snowplow-bucket/shredded/run=2022-01-28-16-30-00/

Our algorithm fails to figure out the bucket is processed already.

Details: https://discourse.snowplowanalytics.com/t/upgrade-shredder-from-0-19-0-to-2-0-0-decoding-failure-in-shredding-job/6261/21

chuwy avatar Jan 27 '22 17:01 chuwy