resdata icon indicating copy to clipboard operation
resdata copied to clipboard

Accessing grdecl grid cells without allocating memory for the whole file?

Open bartoli opened this issue 4 years ago • 1 comments

I tried libecl to access grdecl files from C/C++.

From grdecl files, i need to extract the cell corners coordinates, to find wether a cell is present or not, and the values for some properties for each cell. But i am having issues of excessive memory consumption. Trying to open a 4GB grdecl file is crashing my computer (16GB RAM) with too big allocs (grid ix about 500x500x500).

What i do is

  • Load the various properties using ecl_kw_fscanf_alloc_grdecl_dynamic__(). This stores values in binary format, but uncompresses the '5*-9999' elements, so the buffer for the COORD, ACTNUM, & ZCORN is still 4GB in memory
  • call ecl_grid_alloc_GRDECL_kw() to create an ecl_grid_type. This crashes the computer for the big file.

As i understand, this will always allocate a buffer of nxnynz cells. And the structure for a cell is about 250 bytes each. For a 500x500x500 grid, this only already needs 31GB or RAM.

I don't even need to have all cells in memory at a given time. I could just go with "cell iterator", that from raw ecl_kw_type buffers, gives me access to the corner coordinates for each cell sucesssive cell, so i can store them in an other format. Is there a way to access the cells geometry information and properties values without the need to have this much data in memory?

bartoli avatar Aug 03 '20 15:08 bartoli

Hi @bartoli

Unfortunately I don't believe there is an option to do anything other than to populate the entire grid model on instantiation.

I agree that it would be useful to be able to lazily load parts of the grid though, but unfortunately it isn't anything that is on our priority list.

lars-petter-hauge avatar Aug 06 '20 10:08 lars-petter-hauge