root icon indicating copy to clipboard operation
root copied to clipboard

Overcome 1GB size limit for IO buffers

Open sawenzel opened this issue 4 years ago • 2 comments

Explain what you would like to see improved

ROOT currently has a size limitation (~1GB) for the number of bytes that can go into a single entry of a TBranch. This limitation is due to using code based on 32bit integers in the IO mechanism.

For Run3, the ALICE experiment is turning from an event (singe collision) - based data processing to a processing based on event-collections (timeframes). A timeframe becomes the basic data index. In consequence this puts us into a situation in which we often need to store/serialize very large amounts of data per TBranch entries.

So far, we have worked around this by either giving up on ROOT serialization features, or writing specialized IO containers. But it is foreseeable that the problem aggravates and a fix in the ROOT ecosystem would be a much better solution.

Optional: share how it could be improved

Please improve on the ~1GB size limitation, which is does not correspond to modern memory capabilities and 64bit processing. As a short term solution, gaining a factor of 2 would already help us. This might be possible by reviewing the use of int32 vs uint32 etc.

sawenzel avatar Oct 30 '20 14:10 sawenzel