joerd icon indicating copy to clipboard operation
joerd copied to clipboard

EU AWS S3 bucket

Open StrandedKitty opened this issue 1 year ago • 2 comments

We've encountered a problem with slow response times when fetching tiles from the US-based public bucket elevation-tiles-prod (https://registry.opendata.aws/terrain-tiles/) using HTTP. It often takes ~100ms to fetch a single tile. We think that the main problem is that this bucket is hosted in the United States while a lot of our users are from the EU.

There's a mention of a EU-based mirror bucket (elevation-tiles-prod-eu) but it doesn't work. It's not possible to access it neither using HTTP nor S3.

Is it possible by any chance for us to access the dataset and mirror it ourselves? We've tried mirroring this public bucket by moving tiles 1-by-1, but unfortunately there's so many small files that it's not feasible to do it like this.

StrandedKitty avatar Feb 29 '24 09:02 StrandedKitty

At this time we only offer the single bucket in the one region. You're welcome to copy the data over to another bucket in another region if needed. As you found, there are lots of files, but if you parallelize the process it should be possible to copy them over in a few days. AWS also offers tools to migrate data between S3 buckets that you could look into. The bucket is public, so you should already have the access you need to perform the mirror.

iandees avatar Feb 29 '24 15:02 iandees

@iandees Thanks for the quick response! What's the total size of the bucket?

StrandedKitty avatar Feb 29 '24 16:02 StrandedKitty