reading from memory instead of disk?

[sbailey]

Does fitsio support the ability to read from memory instead of disk?

This question is the inverse of #270, which documents how to write to memory instead of disk.

e.g. I'm looking for something like

fx = fitsio.FITS(io.BytesIO(data), 'r')  #- doesn't work

equivalent to

fx = astropy.io.fits.open(io.BytesIO(data))

Is this possible with fitsio?

[esheldon]

I don't think this particular things works (io.BytesIO(data)) , but I think cfitsio can read from memory. There may be a way to make it happen

[beckermr]

Do we need to pass down some emulated interface to a C file? That might be a tall order.

[esheldon]

Yeah, I'm not saying I know how to do it.

[beckermr]

https://heasarc.gsfc.nasa.gov/docs/software/fitsio/c/c_user/node91.html

This looks annoying and not fun.

[beckermr]

there is also mem:// here: https://heasarc.gsfc.nasa.gov/docs/software/fitsio/c/c_user/node85.html

[sbailey]

Thanks. My interest for this in-memory reading isn't urgent, but I just wanted to check if I was missing some pre-existing mem:// like trick to make it work analogous to the writing case. FWIW, use cases I was considering:

• pulling FITS contents from a non-posix object store as a binary blob and then parsing it as a FITS file in memory without having to pass it via an actual file on disk
• working on a filesystem with good bandwidth but poor/variable latency on individual I/O operations such that doing a single big read and then doing the parsing / seek / etc. in memory might be better performance. Maybe

[beckermr]

working on a filesystem with good bandwidth but poor/variable latency on individual I/O operations such that doing a single big read and then doing the parsing / seek / etc. in memory might be better performance. Maybe

So cfitsio does a lot of buffering internally. For this specific purpose, you may find that you don't need to buffer in memory yourself as well.