gdal
gdal copied to clipboard
OSGeo
Error while reading geotiff: IReadBlock failed at X offset *, Y offset *: TIFFReadEncodedTile() failed
Expected behavior and actual behavior.
I expect to be able to read a specific location of a geotiff but I'm getting an error saying "IReadBlock failed at X offset *, Y offset *: TIFFReadEncodedTile() failed.".
I have seen this issue for many years in many different versions, but I now finally have a dataset on which I can reproduce the issue. I can provide the data privately if it helps to fix the problem.
Steps to reproduce the problem.
The issue can be reproduced in C# with the code below. This is only one specific location where I know it crashes, but there are multiple.
RasterIOExtraArg extraArg = new RasterIOExtraArg();
extraArg.nVersion = 1;
extraArg.eResampleAlg = RIOResampleAlg.GRIORA_CubicSpline;
extraArg.dfXOff = 49265.901653834488;
extraArg.dfYOff = 665.27952667756563;
extraArg.dfXSize = 129.00000000002183;
extraArg.dfYSize = 129.000000000015;
extraArg.bFloatingPointWindowValidity = 1;
float[] byteBuffer = new float[16384];
int[] bandMap = { 1 };
Dataset myDataset = Gdal.Open([MY_FILENAME], Access.GA_ReadOnly)
myDataset .ReadRaster(
xOff = 49265,
yOff = 665,
xSize = 130,
ySize = 130,
byteBuffer,
buf_xSize = 128,
buf_ySize = 128,
bandCount = 1,
bandMap = bandMap,
pixelSpace = 0,
lineSpace = 0,
bandSpace = 0,
extraArg = extraArg);
It can also be reproduced in QGIS by trying to get the pixel info in the same location. I can provide data privately so you can easily find the location. You should get an error like in the screenshot below.
Operating system
Windows 11 - 64 bit (OS build 22000.556)
GDAL version and provenance
GDAL 3.3.3 in C# from https://www.gisinternals.com/ It can also be reproduced in QGIS 3.10.8 (which uses GDAL 3.0.4) The first time I have encountered this issue must have been more than 4 years ago so I assume it happens in all versions of GDAL.
Try if you can reproduce the error with a raster that you can create with gdal_create https://gdal.org/programs/gdal_create.html
gdal_create -if your_raster.tif test_raster.tif
If you can it should be safe to share the new file, it is empty.
This might be a usage issue: the difference between extraArg.dfXSize and buf_xsize (same for y) is >= 1 pixel. It should be < 1 pixel, given that the purpose of the floating point values is to be more accurate than the integer values, but they should remain consistent with them. If that doesn't fix the issue, then we'd need either the exact file, or something representative enough that can reproduce the issue as suggested by @jratike80
The issue also happens if I pass 'null' instead of extra args. In QGIS they will surely also have different parameters and it also happens there. So it's most likely not related to the extraArgs.
Through the following link you can download the data: https://www.dropbox.com/t/TgcELr8lZsD5nsFM
- 'ReadIssueRaster.tif' is the dataset that triggers the crash.
- 'issueArea.shp' is marks one area where you can trigger the crash.
Hello I have downloaded your image and I tried to read the tile (190 , 3). With GDAL 3.4.2, RasterIO returns an error code. I tried also with an "home-made" software, and I have also an error in the LZW decompression. The LZW stream seems to be invalid. But I did not have time to really investigate (thus there is perhaps a bug in my software). What software did you use to generate this image ? Do you have a software that is able to read this image ?
The file size of the image makes me think that it could be created as a normal TIFF and then reached the 4 GB size limit here:
ERROR 1: LZWDecode:Wrong length of decoded string: data probably corrupted at scanline 48384.
The header seems to tell if image is TIFF or BigTIFF stackoverflow.com/questions/43376951/tiff-versus-bigtiff but I did not dare to try to interpret the header bytes.
How did you create the file? Did you create it as BigTIFF?
I currently don't have the original data, but I am trying to get it. I do know that it has been processed with GDAL_Translate with the options below. It could be that the original was not BigTiff. Could that be the reason why we see this crash? If the issue really happens because of a corrupt block, that probably means that it's not possible to create a fix in GDAL's read method, right?
-of "GTiff"
-co "BIGTIFF=YES"
-co "TILED=YES"
-co "BLOCKXSIZE=256"
-co "BLOCKYSIZE=256"
-co "NUM_THREADS=ALL_CPUS"
-ot "Float32"
-co "INTERLEAVE=BAND"
-co "COMPRESS=LZW"
-co "PREDICTOR=3"
-b "1"
-q
-stats
I don't believe this is a BigTIFF related issue. One faulty block is (189,1) which is at offset 72524846.
@ktheijs I'm afraid we can't do anything this ticket with the elements curently available. 2 hypothesis:
- there was a corruption of the file after it was generated
- GDAL (more likely the underlying libtiff) produced a corrupted file. But if that was the case, we'd need a reproducing recipee that takes a non-corrupted file as input and generates the corrupt file.
Ok thanks @rouault , that's what I was thinking too. I'll do my best to get you that recipee. It seems harder than I hoped though.
When I search online for that error, people keep mentioning that it might be related to having insufficient memory. In the case that I've shared they are also trying to mosaic over 300GB of data. Do you think that could have something to do with it?
This could also explain why it is so hard to reproduce (as it is device specific).
References: https://lists.osgeo.org/pipermail/gdal-dev/2010-December/026973.html https://legacy.imagemagick.org/discourse-server/viewtopic.php?t=23490 https://www.mail-archive.com/[email protected]/msg09226.html https://community.safe.com/s/question/0D54Q00008JLG8vSAH/error-building-mosaic-from-tiff-files
Maybe https://trac.osgeo.org/osgeo4w/ticket/740 is related to this issue.
I experience the same issue.
0...10..ERROR 1: ZSTDDecode:Error in ZSTD_decompressStream(): Corrupted block detected
ERROR 1: TIFFReadEncodedTile() failed.
ERROR 1: file.tif.ovr.tmp, band 1: IReadBlock failed at X offset 27, Y offset 15: TIFFReadEncodedTile() failed.
.20
But in my case rerunning the same gdal command throws the error for one to a couple of times and then runs fine without error.
I created tiles in R using gdalUtils (which might be the problem)
gdalUtils::gdal_translate(infile,
outfile,
of="GTiff",
co=c("TILED=YES", "COMPRESS=WEBP", "WEBP_LOSSLESS=True", "NUM_THREADS=ALL_CPUS", "COPY_SRC_OVERVIEWS=YES", "DISCARD_LSB=0,0,2", "BLOCKXSIZE=512", "BLOCKYSIZE=512"))
then created a VRT
gdalbuildvrt test.vrt tile1.tif tile2.tif
and run gdal_translate
gdal_translate test.vrt file.tif -ot byte -of COG -co COMPRESS=WEBP -co QUALITY=100 -co RESAMPLING=NEAREST -co SPARSE_OK=TRUE -a_nodata "0 0 0" -co NUM_THREADS=ALL_CPUS -co BIGTIFF=YES
@wiesehahn could you provide input datasets that can be used to reproduce the issue (that you generated the input tiles with R shouldn't be related. I assume that "gdalinfo -checksum tile1.tif" or tile2.tif works fine ? and that the issue is random failures when generating the COG file ?)
gdalinfos results seem to be fine at first glance. here are two tiles and also the corresponding VRT
here are two tiles and also the corresponding VRT
I've tried to reproduce with those files, on Linux & Windows with latest master on release builds, and that works consistently fine, generating the same output file each time. 12 virtual CPUs here (Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz with 6 cores and hyperthreading)
Hi, I am working with QGIS 3.26.0 on Windows 10 pro, I am experiencing this error message: "crn25_109: IReadBlock failed at X offset 10, Y offset 0: TIFFReadEncodedTile() failed." I can load and see the file anyway, but I have this annoying error message, error message pops up only with this tif file. Same error was with previous QGIS releases. Here you can download the zip file from local governative site, EPSG is 6708. https://irdat.regione.fvg.it/cartogdownload/CRN_RASTER/6708/crn25_109_RDN2008_TM33.zip
Hi, I am working with QGIS 3.26.0 on Windows 10 pro, I am experiencing this error message: "crn25_109: IReadBlock failed at X offset 10, Y offset 0: TIFFReadEncodedTile() failed." I can load and see the file anyway, but I have this annoying error message, error message pops up only with this tif file. Same error was with previous QGIS releases. Here you can download the zip file from local governative site, EPSG is 6708. https://irdat.regione.fvg.it/cartogdownload/CRN_RASTER/6708/crn25_109_RDN2008_TM33.zip
Not reproducable OSGeo4W (QGIS 3.26.0 w/ GDAL 3.5.0). No error - just "No transformation between IMAGINE GeoTIFF Support ERDAS IMAGINE 2016 16.0.0.318 Geocoding information not available Projection Name = Unknown Units = meters GeoTIFF Units = meters and EPSG:4326 available" (translated from german)
Not reproducable OSGeo4W (QGIS 3.26.0 w/ GDAL 3.5.0).
@jef-n, trying to zoom in and out in the northern part of the image:
2022-06-29T16:29:19 WARNING crn25_109_RDN2008_TM33 — crn25_109.tif : IReadBlock failed at X offset 1, Y offset 0: TIFFReadEncodedTile() failed.
2022-06-29T16:29:33 WARNING crn25_109_RDN2008_TM33 — crn25_109.tif : IReadBlock failed at X offset 26, Y offset 0: TIFFReadEncodedTile() failed.,IReadBlock failed at X offset 26, Y offset 0: TIFFReadEncodedTile() failed.,IReadBlock failed at X offset 26, Y offset 0: TIFFReadEncodedTile() failed.
2022-06-29T16:29:33 WARNING crn25_109_RDN2008_TM33 — crn25_109.tif : IReadBlock failed at X offset 26, Y offset 0: TIFFReadEncodedTile() failed.
2022-06-29T16:29:39 WARNING crn25_109_RDN2008_TM33 — crn25_109.tif : IReadBlock failed at X offset 16, Y offset 0: TIFFReadEncodedTile() failed.,IReadBlock failed at X offset 16, Y offset 0: TIFFReadEncodedTile() failed.,IReadBlock failed at X offset 16, Y offset 0: TIFFReadEncodedTile() failed.
2022-06-29T16:29:39 WARNING crn25_109_RDN2008_TM33 — crn25_109.tif : IReadBlock failed at X offset 16, Y offset 0: TIFFReadEncodedTile() failed.
2022-06-29T16:29:53 WARNING crn25_109_RDN2008_TM33 — crn25_109.tif : IReadBlock failed at X offset 35, Y offset 0: TIFFReadEncodedTile() failed.,IReadBlock failed at X offset 35, Y offset 0: TIFFReadEncodedTile() failed.,IReadBlock failed at X offset 35, Y offset 0: TIFFReadEncodedTile() failed.
2022-06-29T16:29:53 WARNING crn25_109_RDN2008_TM33 — crn25_109.tif : IReadBlock failed at X offset 35, Y offset 0: TIFFReadEncodedTile() failed.
2022-06-29T16:30:19 WARNING crn25_109_RDN2008_TM33 — crn25_109.tif : IReadBlock failed at X offset 6, Y offset 0: TIFFReadEncodedTile() failed.,IReadBlock failed at X offset 6, Y offset 0: TIFFReadEncodedTile() failed.,IReadBlock failed at X offset 6, Y offset 0: TIFFReadEncodedTile() failed.
2022-06-29T16:30:19 WARNING crn25_109_RDN2008_TM33 — crn25_109.tif : IReadBlock failed at X offset 6, Y offset 0: TIFFReadEncodedTile() failed.
2022-06-29T16:30:23 WARNING crn25_109_RDN2008_TM33 — crn25_109.tif : IReadBlock failed at X offset 4, Y offset 0: TIFFReadEncodedTile() failed.,IReadBlock failed at X offset 4, Y offset 0: TIFFReadEncodedTile() failed.,IReadBlock failed at X offset 4, Y offset 0: TIFFReadEncodedTile() failed.
2022-06-29T16:30:23 WARNING crn25_109_RDN2008_TM33 — crn25_109.tif : IReadBlock failed at X offset 4, Y offset 0: TIFFReadEncodedTile() failed.
https://irdat.regione.fvg.it/cartogdownload/CRN_RASTER/6708/crn25_109_RDN2008_TM33.zip
as far as I can see, the full resolution image is OK, but the first overview level is corrupted. The issue is on the data producer side.
I have experienced this error after attempting to open an image that has been downloaded from S3 using s3fs. The same image can be read from S3 direct without error, so the file must be corrupted. Strangely it is only affecting one band
I also experienced this error after attempting to open an Landsat image. The data is just official Level 2 product. My computer environment is Windows 10 ，python=3.8， GDAL=3.6.0，but on my other computer，GDAL=3.3.3, it can work. @rouault
I think the issue is a GDAL bug. I created a new anaconda environment set the same packages except GDAL version, when GDAL version > 3.3.3, the bug will happen. @rouault
This is official Level 2 product https://drive.google.com/file/d/1Z02jXPoO8qNKorygflv9sdwdkTYDep-V/view?usp=sharing,I also find that if GDAL is installed by conda ,the bug will happen(No matter which version), but if GDAL is installed by whl file ,it can work, the version also should be < 3.3.*. I download the whl file from https://www.lfd.uci.edu/~gohlke/pythonlibs/#gdal @jratike80
@Zhugguanglei, I don't see any issue opening the provided raster with QGIS 3.28.5 using GDAL 3.6.3 or QGIS 3.28.6 using GDAL 3.6.4.
Works for me with GDAL 3.7.0dev-5ce7ba068f, released 2023/01/17 on Windows, from gisinternals, and with GDAL 3.6.4, released 2023/04/17 from OSGeo4W. Tested with this command that I suppose to read and copy the full resolution data and all overviews as well:
gdal_translate -of COG LC08_L2SP_128032_20200603_20200824_02_T1_SR_B4.TIF copy2.tif --config COPY_SRC_OVERVIEWS YES
gdal_translate -of COG LC08_L2SP_128032_20200603_20200824_02_T1_SR_B4.TIF copy2.tif --config COPY_SRC_OVERVIEWS YES
Works also for me， tested with your command ，but when I run the code：
tif_file = r'D:\Desktop\Work\2022\smart_irrigation\LC08_L2SP_128032_20200603_20200824_02_T1_SR_B4.TIF' in_ds = gdal.Open(tif_file ) in_band = in_ds.GetRasterBand(1) ds_arr = in_band.ReadAsArray()
when I use the pycharm debug the ds_rr shows is None and it produced the following information：
ERROR 1: ZIPDecode:Decoding error at scanline 1536 ERROR 1: TIFFReadEncodedTile() failed. ERROR 1: D:\Desktop\Work\2022\smart_irrigation\LC08_L2SP_128032_20200603_20200824_02_T1_SR_B4.TIF, band 1: IReadBlock failed at X offset 6, Y offset 0: TIFFReadEncodedTile() failed.
@jratike80
@Zhugguanglei, using the provided code, in my system with GDAL 3.6.3 (Windows - OSGeo4W), the ds_array variable is not None:
from osgeo import gdal
in_ds = gdal.Open(tif_file)
in_band = in_ds.GetRasterBand(1)
ds_arr = in_band.ReadAsArray()
print( ds_arr.shape, ds_arr.dtype, ds_arr.min(), ds_arr.max() ) # -> (7981, 7851) uint16 0 55243
Because the error comes from ZIPDecode: I wonder if there could be some difference in the library that takes care of decompressing the DEFLATE compressed data.
@Zhugguanglei Trying with a fresh Conda environment on Windows with the conda-forge channel, I can't reproduce any issue.
As this is deflate TIFF compression, the relevant libraries are libgdal, libtiff, zlib and libdeflate. Here are the versions I've tested (as indicate by "conda list"):
# Name Version Build Channel
libgdal 3.6.4 h36c8192_2 conda-forge
libtiff 4.5.0 h6c8260b_6 conda-forge
zlib 1.2.13 hcfcfb64_4 conda-forge
libdeflate 1.18 hcfcfb64_0 conda-forge
Because the error comes from
ZIPDecode:I wonder if there could be some difference in the library that takes care of decompressing the DEFLATE compressed data.
Thanks for your reply, will try your suggestion