TileDB
TileDB copied to clipboard
Out-of-core fragment metadata reading
Currently we load all metadata of a fragment in main memory. For large arrays, the MBRs and other information about tiles may become a large memory issue, especially when multiple independent processes run on the same machine (which do not share a common fragment metadata cache).
We should transition to a solution where certain fragment metadata (like MBRs) are kept in persistent storage and read in an out-of-core fashion on demand, in order to minimize the memory footprint of the fragment metadata.