Ingest: Add geotiff support

[raprasad]

trafficstars

For GeoTIFF files added to Dataverse:

1. Identify the file as a GeoTIFF and make an appropriate thumbnail.
   • http://gis.stackexchange.com/questions/18672/how-to-determine-if-a-tiff-is-georeferenced-or-not
2. Consider adding WorldMap support/visualization. (WorldMap already supports GeoTIFF

[eunices]

Hi, would like to ask if there are any updates on this issue?

geotiff file thumbnails are broken. This issue only applies to some geotiff but not all (see https://researchdata.ntu.edu.sg/dataverse/ASE?q=tiff&types=dataverses%3Adatasets%3Afiles&sort=score&order=desc&page=1).

---

A colleague replicated it on the demo.dataverse.org instance.
Two geotiff files with this issue were uploaded and thumbnails associated with the file are broken.

Dataset uploaded on: https://demo.dataverse.org/dataset.xhtml?persistentId=doi:10.70122/FK2/TV3OWZ Files involved: middle_lower_Amazon_bathymetry_v2.tiff, Table2_Sediment.tif

The thumbnail is 0 bytes according to Jim Myers on https://groups.google.com/g/dataverse-community/c/K9eUCw8erKc.

Both files are geotiff, verified with gdal

(base) D:_dev\dataverse>gdalinfo data-issue-20210830/middle_lower_Amazon_bathymetry_v2.tiff Driver: GTiff/GeoTIFF Files: data-issue-20210830/middle_lower_Amazon_bathymetry_v2.tiff Size is 7316, 2005 Coordinate System is: GEOGCRS["WGS 84", DATUM["World Geodetic System 1984", ELLIPSOID["WGS 84",6378137,298.257223563, LENGTHUNIT["metre",1]]], PRIMEM["Greenwich",0, ANGLEUNIT["degree",0.0174532925199433]], CS[ellipsoidal,2], AXIS["geodetic latitude (Lat)",north, ORDER[1], ANGLEUNIT["degree",0.0174532925199433]], AXIS["geodetic longitude (Lon)",east, ORDER[2], ANGLEUNIT["degree",0.0174532925199433]], ID["EPSG",4326]] Data axis to CRS axis mapping: 2,1 Origin = (-59.898000000000003,-1.742771843488000) Pixel Size = (0.000903557453000,-0.000903557453000) Metadata: AREA_OR_POINT=Area Image Structure Metadata: COMPRESSION=LZW INTERLEAVE=BAND Corner Coordinates: Upper Left ( -59.8980000, -1.7427718) ( 59d53'52.80"W, 1d44'33.98"S) Lower Left ( -59.8980000, -3.5544045) ( 59d53'52.80"W, 3d33'15.86"S) Upper Right ( -53.2875737, -1.7427718) ( 53d17'15.27"W, 1d44'33.98"S) Lower Right ( -53.2875737, -3.5544045) ( 53d17'15.27"W, 3d33'15.86"S) Center ( -56.5927868, -2.6485882) ( 56d35'34.03"W, 2d38'54.92"S) Band 1 Block=128x128 Type=Float32, ColorInterp=Gray Min=1.764 Max=10.351 Minimum=1.764, Maximum=10.351, Mean=5.436, StdDev=1.919 NoData Value=-3.4028234663852886e+38 Metadata: RepresentationType=ATHEMATIC STATISTICS_COVARIANCES=3.684419636731698 STATISTICS_MAXIMUM=10.35097694397 STATISTICS_MEAN=5.4357336787474 STATISTICS_MINIMUM=1.7639776468277 STATISTICS_SKIPFACTORX=1 STATISTICS_SKIPFACTORY=1 STATISTICS_STDDEV=1.9194842111181

(base) D:_dev\dataverse>gdalinfo data-issue-20210830/Table2_Sediment.tif Driver: GTiff/GeoTIFF Files: data-issue-20210830/Table2_Sediment.tif Size is 274, 170 Coordinate System is: PROJCRS["CGCS2000_3_Degree_GK_CM_114E", BASEGEOGCRS["China Geodetic Coordinate System 2000", DATUM["China 2000", ELLIPSOID["CGCS2000",6378137,298.257222101004, LENGTHUNIT["metre",1]]], PRIMEM["Greenwich",0, ANGLEUNIT["degree",0.0174532925199433]], ID["EPSG",4490]], CONVERSION["Transverse Mercator", METHOD["Transverse Mercator", ID["EPSG",9807]], PARAMETER["Latitude of natural origin",0, ANGLEUNIT["degree",0.0174532925199433], ID["EPSG",8801]], PARAMETER["Longitude of natural origin",114, ANGLEUNIT["degree",0.0174532925199433], ID["EPSG",8802]], PARAMETER["Scale factor at natural origin",1, SCALEUNIT["unity",1], ID["EPSG",8805]], PARAMETER["False easting",500000, LENGTHUNIT["metre",1], ID["EPSG",8806]], PARAMETER["False northing",0, LENGTHUNIT["metre",1], ID["EPSG",8807]]], CS[Cartesian,2], AXIS["easting",east, ORDER[1], LENGTHUNIT["metre",1]], AXIS["northing",north, ORDER[2], LENGTHUNIT["metre",1]], ID["EPSG",4547]] Data axis to CRS axis mapping: 1,2 Origin = (474514.743200000375509,2527758.696948537603021) Pixel Size = (250.000000000000000,-250.000000000000000) Metadata: AREA_OR_POINT=Area Image Structure Metadata: INTERLEAVE=BAND Corner Coordinates: Upper Left ( 474514.743, 2527758.697) (113d45' 6.10"E, 22d50'53.86"N) Lower Left ( 474514.743, 2485258.697) (113d45' 8.58"E, 22d27'52.25"N) Upper Right ( 543014.743, 2527758.697) (114d25' 8.74"E, 22d50'52.57"N) Lower Right ( 543014.743, 2485258.697) (114d25' 4.55"E, 22d27'50.98"N) Center ( 508764.743, 2506508.697) (114d 5' 7.00"E, 22d39'23.67"N) Band 1 Block=128x128 Type=Float32, ColorInterp=Gray Min=0.000 Max=688.500 Minimum=0.000, Maximum=688.500, Mean=26.463, StdDev=32.489 NoData Value=-3.4028234663852886e+38 Metadata: RepresentationType=ATHEMATIC STATISTICS_MAXIMUM=688.5 STATISTICS_MEAN=26.463243982434 STATISTICS_MINIMUM=0 STATISTICS_SKIPFACTORX=1 STATISTICS_SKIPFACTORY=1 STATISTICS_STDDEV=32.488883884369

[qqmyers]

There are a lot of things that could be done. At a minimum, Dataverse shouldn't send 0 byte thumbnails and should use the default icon instead. More usefully, I found that ImageMagick and GraphMagick (#8014), with the latter producing a clearer thumb in my quick test, can create a png thumb from these files, so using it instead of the jai-imageio library in these cases (when jai has an error/produces 0 bytes) could help. (FWIW: I also tried updating to the latest jai version - that didn't help).

[PaulBoon]

Hi, got the same problem on our dataverse.nl instance (v5.6), where a recently published dataset with geoTiff files destroys the layout of the root verse (home) page which is annoying.

When uploading one of this geoTiff files on a 5.10.1 version it seems to be fixed, but I can't find any info about that.

Anyways, the only thing that helped was overwriting the tumbs with another image. Only did that for the datasets (thumb48) which need a 48x48 PNG image which I took from a screendump. Maybe others find this useful.

After upgrading to 5.10.1 is there a way to force regeneration of the thumbs via an API, or should I just remove them on disk @pdurbin ?

[pdurbin]

@PaulBoon a few things...

There are some APIs for managing thumbnails but they aren't documented. Please feel free to create an issue about this! For now, please see https://groups.google.com/g/dataverse-community/c/goFs0D61es4/m/B4pLV-_lAwAJ

There are a few related issues you might be interested in:

• #8167
• #8664

With regard to removing the thumbnails on disk, if it works, please go for it! Thumbnails are regenerated when they don't exist. You can even generate thumbnails of arbitrary size. See imageThumb=64 at https://guides.dataverse.org/en/5.11.1/api/dataaccess.html

[mreekie]

grooming:

• discussed in NetCDF meeting.
• not in the queue. Not adding tag.