parse_gctx takes too much memory when we inquire specific columns and rows

[Dom303]

When loading a very large gctx file ~24Gb on my laptop with 16 Gb using the function cmapPy.pandasGEXpress.parse.parse, I run out of memory with the following error:

Unable to allocate array with shape (473647,) and data type

If I use cidx to select a very low number of columns, then there is no more error. However, when I request certain columns and certain rows, using both cidx and ridx, the same allocation error occurs. This indicates that the row filtering is applied, followed by the column filtering. This is a bad behaviour when dealing with very large cmap files, where it would be preferable that both filtering be applied simultaneously to avoid running out of RAM.

The problem comes from pandasGEXpress.parse_metadata_df, at the line curr_dset.read_direct(temp_array). The function read_direct simply reads all the rows/columns without any means of filtering.

[dhamelse]

I am also experiencing this issue.

[mark-liddell]

This is also causing issues for me