Read batches of rows from HDU tables?

[Theodlz]

Hi!

I am trying to read from large-ish datasets. In the docs, I see there is a method to read a single row so I tried using that in a row to extract all my data. It's very very slow that way, much slower than if my data was stored in a simple CSV or parquet, for which I can get iterators to loop over entries.

Is there a way to efficiently read large tables, that I may have missed?

Thanks!

[simonrw]

You can iterate over columns one at a time with columns() if you are ok to filter out the rows you want yourself. Or you can read_col_range to read a number of rows per column. You could write your own iterator to achieve this quite easily I think.

We don't support fetching multiple rows at once (eg into a Vec<FitsRow> but this would probably not be hard to implement yourself. If you want any help I'm happy to guide however my time for this project is very limited at the moment.