Reading very large .fits.gz file fails.

[rainwoodman]

We (@ekitanidis) were trying to read a 2.1GB .fits.gz file (legacy survey DR8 annotated CCD file, prerelease). The read operation failed with cfitsio internal error.

Seems to be when it is opening the inmemory file for decompression -- could be because the decompressed file is greater than 4GB.

astropy was able to read the same file without a problem. Is this something we can fix?

[esheldon]

This is the same issue as #199

[esheldon]

Where do we stand on this issue? Does it work for the latest version (1.1.3), for which we have updated cfitsio version to 3.49?

[dstndstn]

Coming back to this much later... with a fresh build (1.1.7-2-g59fba1e) I am still getting an error trying to read a large .fits.gz (3.4 GB compressed, 9.1 GB uncompressed):

> python -c "import fitsio; print(fitsio.__file__); fitsio.read('/global/cfs/cdirs/cosmo/work/legacysurvey/dr10/ccds-annotated-dr10-v5.fits.gz')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/global/u1/d/dstn/fitsio2/fitsio/fitslib.py", line 139, in read
    with FITS(filename, **kwargs) as fits:
  File "/global/u1/d/dstn/fitsio2/fitsio/fitslib.py", line 520, in __init__
    self._FITS = _fitsio_wrap.FITS(filename, self.intmode, create)
OSError: FITSIO status = 414: error uncompressing image
failed to uncompress file into memory (compress_open)
failed to find or open the following file: (ffopen)
/global/cfs/cdirs/cosmo/work/legacysurvey/dr10/ccds-annotated-dr10-v5.fits.gz

On the plus side, fitsio can read the uncompressed version just fine.

Let me know if there's any debugging I can do to help. Thanks!

[esheldon]

Can you read subsets of rows or columns?

[dstndstn]

Hi, unfortunately it doesn't look like reading row or column subsets helps; fitsio.read('/global/cfs/cdirs/cosmo/work/legacysurvey/dr10/ccds-annotated-dr10-v5.fits.gz', columns=['expnum']) and fitsio.read('/global/cfs/cdirs/cosmo/work/legacysurvey/dr10/ccds-annotated-dr10-v5.fits.gz', rows=[0]) yield the same error as above.

[esheldon]

I think we might need to report this to the cfitsio maintainers