rasterframes icon indicating copy to clipboard operation
rasterframes copied to clipboard

RasterSource attempts to read hadoop scheme with gdal reader

Open vpipkt opened this issue 6 years ago • 0 comments

Trying to read a hadoop scheme with gdal installed such as s3a://${bucket}/${prefix}.TIF produces an error such as below message/stack trace.

This seems consistent with documented use of HDFS versus GDAL schemes and such: which is the s3n / s3a schemes are not compatibal with GDAL read and default to hadoop read.

However in the stack trace we see that the behavior is to read the s3a scheme URI with the GDAL reader (note GDALRasterSource). Roughly here needs a check for schemes perhaps: https://github.com/locationtech/rasterframes/blob/0.8.0/core/src/main/scala/org/locationtech/rasterframes/ref/RasterSource.scala#L104-L138

stack trace

ERROR 4: landsat-pds/c1/L8/190/028/LC08_L1TP_190028_20190705_20190719_01_T1/LC08_L1TP_190028_20190705_20190719_01_T1_B8.TIF: No such file or directory
2019-08-23 17:39:45 ERROR RasterSourceToRasterRefs:77 - Error fetching data for one of: GDALRasterSource(s3a://landsat-pds/c1/L8/190/028/LC08_L1TP_190028_20190705_20190719_01_T1/LC08_L1TP_190028_20190705_20190719_01_T1_B8.TIF)
java.lang.Exception: get_transform or get_widht_height
	at geotrellis.contrib.vlm.gdal.GDALDataset$.rasterExtent$extension1(GDALDataset.scala:45)
	at geotrellis.contrib.vlm.gdal.GDALDataset$.rasterExtent$extension0(GDALDataset.scala:37)
	at geotrellis.contrib.vlm.gdal.GDALBaseRasterSource$class.rasterExtent(GDALBaseRasterSource.scala:69)
	at geotrellis.contrib.vlm.gdal.GDALRasterSource.rasterExtent$lzycompute(GDALRasterSource.scala:22)
	at geotrellis.contrib.vlm.gdal.GDALRasterSource.rasterExtent(GDALRasterSource.scala:22)
	at geotrellis.contrib.vlm.RasterSource$class.cols(RasterSource.scala:76)
...
...
...

Originally posted by @vpipkt in https://github.com/locationtech/rasterframes/issues/38#issuecomment-524404552

vpipkt avatar Aug 23 '19 18:08 vpipkt