snowplow-rdb-loader icon indicating copy to clipboard operation
snowplow-rdb-loader copied to clipboard

Add support for Snowplow asset buckets mirroring

Open chuwy opened this issue 8 years ago • 2 comments

Cannot promise this will be implemented, @acgray, but what would your implementation look like? Is it another setting in aws.s3.buckets?

chuwy avatar Aug 17 '17 10:08 chuwy

Nothing even that complicated - just use the existing aws.s3.buckets.assets setting to fill the name for the Snowplow hosted assets bucket, instead of the hardcoded value here:

https://github.com/snowplow/snowplow/blob/master/4-storage/rdb-loader/src/main/scala/com/snowplowanalytics/snowplow/rdbloader/ShreddedType.scala#L67

That would knock out your automatic region selection but I'd have thought given aws.s3.buckets.assets is used by other components the user should be updating it for their own region anyway, and perhaps an extra documentation note on that point would be sufficient.

acgray avatar Aug 18 '17 08:08 acgray

Neat idea... Why would it have to knock out the automatic region selection? We could check if aws.s3.buckets.assets == "s3://snowplow-hosted-assets"

alexanderdean avatar Aug 18 '17 10:08 alexanderdean