Europe 25 meter DEM

[nvkelso]

• http://www.eea.europa.eu/data-and-maps/data/eu-dem#tab-metadata

• License: CC-BY

• Geographic coverage:

  • Albania Austria Belgium Bosnia and Herzegovina Bulgaria Croatia Cyprus Czechia Denmark Estonia Finland France Germany Greece Hungary Iceland Ireland Italy Kosovo (UNSCR 1244/99) Latvia Liechtenstein Lithuania Luxembourg Macedonia (FYR) Malta Montenegro Netherlands Norway Poland Portugal Romania Serbia Slovakia Slovenia Spain Sweden Switzerland United Kingdom

• Format: Looks similar to SRTM

[mojodna]

A mirror of these files is in s3://mapzen-eudem/

Transcoded versions are in s3://mapzen-dynamic-tiler-test/EU-DEM/0/

[iandees]

Listing files

curl -s https://www.eea.europa.eu/data-and-maps/data/eu-dem \
  | grep -o '<a href=['"'"'"][^"'"'"']*['"'"'"]' \
  | sed -e 's/^<a href=["'"'"']//' -e 's/["'"'"']$//' \
  | grep entr_r_4258_1_arcsec_gsgrda-eudem-dem-europe_2012_rev1 \
  > eudem_listing.txt

Mirroring files to openterrain S3

cat eudem_listing.txt \
    | xargs \
      -I {} \
      -P 24 \
      -n 1 \
      sh -c 'export f="{}"; export s3="s3://elevation-sources-prod/eudem/`basename $f`"; curl -s $f | AWS_DEFAULT_PROFILE=openterrain aws s3 cp - $s3; echo $s3' \
    tee eudem_s3_objects.txt

… This seems to fail because the download site isn't responding. I'll come back in a bit. …

[iandees]

They're looking into the down-ness:

https://twitter.com/EUEnvironment/status/872430176298303488

[iandees]

Looks to be back up, so I'm doing the mirroring step now.

[iandees]

Transcoding started:

aws s3 ls s3://elevation-sources-prod/eudem/ --recursive | \
grep tif$ | awk '{print $4}' | \
while read filename; do
    bn=$(basename ${filename%.*})
    make submit-job job=aws/transcode-job.json.hbs input=s3://elevation-sources-prod/${filename} output=s3://elevation-sources-transcoded/$(dirname $filename)/${bn} name=${bn:0:50}
done

[iandees]

Some of these jobs are failing because they run out of memory. These are the failures. I'm cloning the jobs through the Batch console and giving them 3GB of memory:

• [x] eudem_dem_5deg_n65e015
• [x] eudem_dem_5deg_n60e025
• [x] eudem_dem_5deg_n50w005
• [x] eudem_dem_5deg_n40e025
• [x] eudem_dem_5deg_n40e000
• [x] eudem_dem_5deg_n35w005
• [x] eudem_dem_5deg_n35e040
• [x] eudem_dem_5deg_n35e035
• [x] eudem_dem_5deg_n35e030