High memory usage by large FileDataIds

[Luzifix]

Hi, when working with high FileDataIds, e.g. 1000000000, a lot of memory is allocated by the resulting large gaps, the FileDataId is used for the calculation of the array size, this result into unnecessarily memory usaged in https://github.com/ladislav-zezula/CascLib/blob/master/src/common/Array.h#L159

What i currently find out: After a FileId is inserted https://github.com/ladislav-zezula/CascLib/blob/master/src/common/FileTree.cpp#L120

The array calculate his size based on the FileId: https://github.com/ladislav-zezula/CascLib/blob/master/src/common/Array.h#L102

The set of m_ItemCountMax is here and the code above is the calculation for the new size: https://github.com/ladislav-zezula/CascLib/blob/master/src/common/Array.h#L199

And when https://github.com/ladislav-zezula/CascLib/blob/master/src/common/Array.h#L159 is called the memory explodes because it trys to allocate m_ItemCountMax * m_ItemSize (In my case is m_ItemCountMax 1073741824 and the m_ItemSize 8 which result in a memory alloucation of 8589934592 bytes

Is there perhaps a way to optimize the allocation of memory and not make it dependent on the FileDataIds?

[ladislav-zezula]

I'll think about it. Are you trying to do it on some specific (or recent) WoW build?

[Luzifix]

I try it with 9.2.0 but i think its not rlly related to a specific version. Just create a RootEntry with a high FileDataId like 1000000000 and you get the same issue

[ladislav-zezula]

I understand, but is this actually a real issue or a theoretical problem that never happens?

[Luzifix]

I noticed the error when I tried to open a RootFile with high custom FileDataIds in https://github.com/TrinityCore/TrinityCore/pull/27993. So its a real issue while modding when working with high FileDataIds.

As workaround i reduce the gap between my Id ranges and blizzard's but the issue is still exists just have with smaller id gap a smaller impact in memory

[ladislav-zezula]

I added a sparse-array object, which maked the memory usage of the FileDataId array somewhat more effective.

64-bit CascView.exe opening the storage of WoW build 31299:

• Old version: 694024 KB of memory
• New version: 675 932 KB of memory

The newer build uses about 18 MB less memory than the older one.

The difference is even larger when you use high file data IDs. Feel free to try the LZ_SparseArraySupport branch in your project. I'll run tests and then publish a new build of CascView.