Ian Dees

We can still have a database, but it doesn't need to be 300+ GB if we can put the bulk data on S3.

(I moved several comments I mistakenly added to this ticket over to https://github.com/tilezen/joerd/issues/165)

#### Listing files ``` curl -s https://www.eea.europa.eu/data-and-maps/data/eu-dem \ | grep -o '

They're looking into the down-ness: https://twitter.com/EUEnvironment/status/872430176298303488

Looks to be back up, so I'm doing the mirroring step now.

#### Transcoding started: ``` aws s3 ls s3://elevation-sources-prod/eudem/ --recursive | \ grep tif$ | awk '{print $4}' | \ while read filename; do bn=$(basename ${filename%.*}) make submit-job job=aws/transcode-job.json.hbs input=s3://elevation-sources-prod/${filename} output=s3://elevation-sources-transcoded/$(dirname...

Some of these jobs are failing because they run out of memory. These are the failures. I'm cloning the jobs through the Batch console and giving them 3GB of memory:...

#### Mirroring data ``` curl -s https://www.boem.gov/webteam/bathymetry/BOEM_Bathymetry_East_meters_tiff.zip | \ AWS_DEFAULT_PROFILE=openterrain aws s3 cp - s3://elevation-sources-prod/boem_gom_bathy/BOEM_Bathymetry_East_meters_tiff.zip curl https://www.boem.gov/webteam/bathymetry/BOEM_Bathymetry_West_meters_tiff.zip | \ AWS_DEFAULT_PROFILE=openterrain aws s3 cp - s3://elevation-sources-prod/boem_gom_bathy/BOEM_Bathymetry_West_meters_tiff.zip ``` #### Transcode data Note...

These both failed during the `rio shapes` call with the following error: ``` CPLE_AppDefinedError: Full reprojection failed, but partial is possible if you define OGR_ENABLE_PARTIAL_REPROJECTION configuration option to TRUE ```

I tried running `rio shapes` on these images locally (outside of Docker) and it failed again. I'm going to double-check that they're out of the footprints database and move on.