snowplow-rdb-loader icon indicating copy to clipboard operation
snowplow-rdb-loader copied to clipboard

AWS streaming transformer should use default chain to set S3 region

Open istreeter opened this issue 3 months ago • 0 comments

In most places snowplow apps let the aws sdk figure out the region using the default chain. But AWS streaming transformer uses hadoop for writing to S3 (only for parquet output format) and infers the region slightly differently.

With this change, we use the AWS SDK to infer the region and then explicitly pass it to hadoop.

This means the transformer should work in other AWS partitions, e.g. aws-china.

istreeter avatar Apr 08 '24 07:04 istreeter