Add 5 m data for Australia

[jdugge]

Geoscience Australia provides a 5 m Digital Elevation Model (DEM) of Australia derived from LiDAR. It's licensed under CC-BY-4.0, dataset covers 245000 km² of "Australia's populated coastal zone; floodplain surveys within the Murray Darling Basin, and individual surveys of major and minor population centres"

[zerebubuth]

Thanks!

[nvkelso]

Looks like there's a new 1m and 2m DEM for New South Wales, see #121.

[iandees]

Data download URLs:

• https://s3-ap-southeast-2.amazonaws.com/elvis.ga.gov.au/elevation/5m-dem/state-mosaics/22be4b55-2465-4320-e053-10a3070a5236_NSW.zip
• https://s3-ap-southeast-2.amazonaws.com/elvis.ga.gov.au/elevation/5m-dem/state-mosaics/22be4b55-2465-4320-e053-10a3070a5236_VIC.zip
• https://s3-ap-southeast-2.amazonaws.com/elvis.ga.gov.au/elevation/5m-dem/state-mosaics/22be4b55-2465-4320-e053-10a3070a5236_QLD.zip
• https://s3-ap-southeast-2.amazonaws.com/elvis.ga.gov.au/elevation/5m-dem/state-mosaics/22be4b55-2465-4320-e053-10a3070a5236_NT.zip
• https://s3-ap-southeast-2.amazonaws.com/elvis.ga.gov.au/elevation/5m-dem/state-mosaics/22be4b55-2465-4320-e053-10a3070a5236_SA.zip
• https://s3-ap-southeast-2.amazonaws.com/elvis.ga.gov.au/elevation/5m-dem/state-mosaics/22be4b55-2465-4320-e053-10a3070a5236_TAS.zip
• https://s3-ap-southeast-2.amazonaws.com/elvis.ga.gov.au/elevation/5m-dem/state-mosaics/22be4b55-2465-4320-e053-10a3070a5236_WA.zip

[iandees]

Listing files

curl -s http://www.ga.gov.au/metadata-gateway/metadata/record/gcat_22be4b55-2465-4320-e053-10a3070a5236 \
  | grep -o '<a href=['"'"'"][^"'"'"']*['"'"'"]' \
  | sed -e 's/^<a href=["'"'"']//' -e 's/["'"'"']$//' \
  | grep state-mosaics \
  | sort \
  | uniq \
  > geoscience_au_listing.txt

Mirroring files

I skipped mirroring because these were already on S3 and are rather large.

cat geoscience_au_listing.txt \
    | xargs \
      -I {} \
      -P 24 \
      -n 1 \
      sh -c 'export f="{}"; export s3="s3://elevation-sources-prod/geoscience_au/`basename $f`"; curl -s $f | AWS_DEFAULT_PROFILE=openterrain aws s3 cp - $s3; echo $s3' \
    tee geoscience_au_s3_objects.txt

Transcoding files

cat geoscience_au_listing.txt | \
while read filename; do
    bn=$(basename ${filename%.*})
   make submit-job job=aws/transcode-job.json.hbs input=${filename} output=s3://elevation-sources-transcoded/geosciences_au/${bn} name=${bn:0:50}
done

[iandees]

These seem to be failing intermittently, probably because the ECS instance is running out of disk space. I'll check these off as they successfully run.

• [ ] NSW
• [x] VIC
• [ ] QLD
• [x] NT
• [ ] SA
• [ ] TAS
• [ ] WA

[iandees]

…these are still failing, even when running on a separate instance with plenty of space. It's not clear to me why they're failing, but it seems that gdal_translate is just stopping at some point:

Processing https://s3-ap-southeast-2.amazonaws.com/elvis.ga.gov.au/elevation/5m-dem/state-mosaics/22be4b55-2465-4320-e053-10a3070a5236_NSW.zip to s3://elevation-sources-transcoded/geosciences_au/22be4b55_NSW...
Downloading https://s3-ap-southeast-2.amazonaws.com/elvis.ga.gov.au/elevation/5m-dem/state-mosaics/22be4b55-2465-4320-e053-10a3070a5236_NSW.zip...
Transcoding...
Transcoding zip:///tmp/tmp.DXfBGJtkdW.22be4b55-2465-4320-e053-10a3070a5236_NSW.zip!NSW/z54/mdbaz54ag.tif
Transcoding bands...
Input file size is 84000, 104800
0...10.delete: s3://elevation-sources-transcoded/geosciences_au/22be4b55_NSW.tif
delete: s3://elevation-sources-transcoded/geosciences_au/22be4b55_NSW.tif.msk
delete: s3://elevation-sources-transcoded/geosciences_au/22be4b55_NSW_footprint.json
delete: s3://elevation-sources-transcoded/geosciences_au/22be4b55_NSW.vrt
delete: s3://elevation-sources-transcoded/geosciences_au/22be4b55_NSW_thumb.png
delete: s3://elevation-sources-transcoded/geosciences_au/22be4b55_NSW.json

[mojodna]

That behavior is consistent with things I've seen when there hasn't been enough memory available. GDAL tries to be sensible about what it's using and seems to end up never finishing. This is why it's wrapped with timeout in places. Try increasing the amount of memory available?

[iandees]

@mojodna I'm doing this on a machine with 64GB of memory and don't see any memory pressure while I'm running it. I wonder if Docker limits memory usage by default?

[mojodna]

I don't think it does, so maybe this is something else... Hmm.

[iandees]

After trying this again on a bare EC2 instance I tried downloading a DEM to my laptop. Unzipping the file that got downloaded with the MacOS X expander created a .cpgz file instead of the expected .tif, so I tried it from the command line and got:

$ unzip 22be4b55-2465-4320-e053-10a3070a5236_TAS.zip
Archive:  22be4b55-2465-4320-e053-10a3070a5236_TAS.zip
   skipping: TAS/tasz55.tif          need PK compat. v4.5 (can do v2.1)
  inflating: TAS/Ancillary/metadata/Digital Elevation Model (DEM) 5 Metre Grid of Australia derived from LiDAR.pdf

So... maybe it's just a broken zipfile?

[nvkelso]

This data is all over the place and missing most of Australia. Since there is no good, easily accessible file for Australia let's travel over to New Zealand where LINZ is straight forward.

[OliverColeman]

Is this project still active? Is there any plan for getting better resolution data for Australia at present?

[iandees]

It's active, but the Earth doesn't change all that much so we don't update it frequently.

Some Australian data made it into the most recent build, but the data we found here didn't cover the entire country. I think there's just a few small pieces that we found. We'd love to see sources for better data in Australia!

[andrewharvey]

As far as I can tell from http://elevation.fsdf.org.au

NSW provides state wide ranging from 1m-5m CC BY. TAS is not under an open license. VIC provides state wide 10m CC BY (http://www.delwp.vic.gov.au/vicmap) and coastal areas 0.5m-1m is not under an open license. QLD not under an open license

I've been doing the clip and ship process for about 1/3rd of NSW, and have this sitting in a private Digital Ocean Space, which I can share if anyone is interested in bypassing the Clip and Ship process.

[OliverColeman]

Does this not provide 5m coverage for all of Australia that is CC BY? http://gaservices.ga.gov.au/site_9/rest/services/DEM_LiDAR_5m/MapServer

[iandees]

It's not easy to get bulk data out of an Esri REST map service. I think that service uses the data mentioned on this ticket above that doesn't successfully unzip.

I'm downloading the data from here: https://data.gov.au/dataset/ds-ga-22be4b55-2465-4320-e053-10a3070a5236/details / https://ecat.ga.gov.au/geonetwork/srv/eng/catalog.search#/metadata/89644 (which may or may not be an updated version of what I linked to above). Once I get it downloaded I'll see if the zip files are still broken.