Add new v1 Alaska DEM data

[nvkelso]

2 meter strip DEM data for Alaska (or 5 meter composite), hot off the presses. #thanksobama

Because Alaska is so far north, in web Mercator space this amounts to A LOT OF TILES to update.

• Press release
• Data portal

License and Usage

ArcticDEM data is an unlicensed product and may be used, distributed, and modified without permission.

It looks like there are known holes in data coverage that are scheduled to be filled.

ArcticDEM data encompasses all land area north of 60° north latitude. In addition, coverage will include all territory of Greenland, the entire state of Alaska, and the Kamchatka Peninsula of the Russian Federation. The project commenced with Alaska and is expected to create a comprehensive elevation model of the Arctic within the two-year term of the U.S. chairmanship of the Arctic Council, which began in April 2015.

Here near Anchorage and Sarah Palin's Russian suburbs cottage:

But at it's highest quality it looks like:

[iandees]

There is a lot of data here. I was going to start mirroring the 2m strips to the sources bucket, but there are well over 30k images, each in the 100's to 1000's of MB each. I'm still trying to understand what's in each of these, but it might make more sense to use the 5m mosaic tiled product they have. That's still a lot of data, but there's (a) much less data and (b) less overlap for us to process.

[nvkelso]

And considering our zoom 15 is nominally 3 meters I think that's totally fine :)

On Jun 5, 2017, at 19:51, Ian Dees [email protected] wrote:

There is a lot of data here. I was going to start mirroring the 2m strips to the sources bucket, but there are well over 30k images, each in the 100's to 1000's of MB each. I'm still trying to understand what's in each of these, but it might make more sense to use the 5m mosaic tiled product they have. That's still a lot of data, but there's (a) much less data and (b) less overlap for us to process.

― You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[iandees]

Going with the 5m data.

Listing files

lftp \
    -c 'find' \
    ftp://ftp.data.pgc.umn.edu/elev/dem/setsm/ArcticDEM/mosaic/v2.0/ \
  | grep tar.gz > pgdc_5m_listing.txt

Mirroring files to openterrain S3

cat pgdc_5m_listing.txt \
    | xargs \
      -I {} \
      -P 24 \
      -n 1 \
      sh -c 'export f="{}"; export s3="s3://elevation-sources-prod/pgdc_5m/`basename $f`"; curl -s $f | AWS_DEFAULT_PROFILE=openterrain aws s3 cp - $s3; echo $s3' \
    tee pgdc_5m_s3_objects.txt

[iandees]

Mirroring is in progress.

[iandees]

Transcoding files

aws s3 ls s3://elevation-sources-prod/pgdc_5m/ --recursive | \
grep tar.gz$ | awk '{print $4}' | \
while read filename; do
    bn=$(basename ${filename%.tar.gz})
    make submit-job job=aws/transcode-job.json.hbs input=s3://elevation-sources-prod/${filename} output=s3://elevation-sources-transcoded/$(dirname $filename)/${bn} name=$(sed 's/\./_/' <<< $bn)
done