dfinity
Big jumps in memory utilization with EOP, Possible memory leak?
I recently started upgrading some of our canisters to EOP, and in the past month we've seen some unexpected increases in memory utilization on one of our canisters, the transaction history canister.
Before EOP, this canister's memory grew gradually and held roughly 30MB of heap.
Then, after upgrading to EOP, the heap size instantly doubled to 64MB, which is expected, as the canister was migrated from 32 bit to 64 bit wasm.
However, over the past month we've seen the heap balloon in size, completely decoupled from the number of transactions in this canister which we believe might be due to a memory leak.
Here are the big jumps in memory.
June 5th, jumps to 192 MB of heap (128 MB heap allocated, 3X in size)
June 27th, jumps to 256 MB (~64 MB more heap allocated)
June 29th, jumps to 320 MB (~64 MB more heap allocated)
As a sanity check, CycleOps has performed 29300 transactions since its inception (29.3k records in the BTree store).
3604 transactions have occurred since June 1st, 2025. This should not result in a 5X in memory (64MB -> 320MB).
Good that you raise this. For Motoko team: I believe this is due the partition size of 64 MB, when it triggers GC then it needs some 1-2 more such partitions. Or maybe a bulk allocation (probably not). Maybe it could be good to lower partition size?
@ByronBecker: What is the Motoko heap size?
@ByronBecker: What is the Motoko heap size?
Prim.rts_heap_size -> 53_696_920 bytes
Prim.rts_memory_size -> 335_544_320 bytes
@alexandru-uta Here's data from the past few days (since July 3rd). I'm pulling rts_heap_size and rts_memory_size stats from it once every 6 hours.
https://docs.google.com/spreadsheets/d/1u08v0OE0zPznLNMniqvXD_JgDW_TYdWcj0KvAE2QCp4/edit?usp=sharing
It seems like the heap allocations (although large) might be working as expected, but the motoko heap allocation for the BTree Map in an EOP canister is growing at a pretty good clip - much quicker than pre-EOP.
Between July 3rd - July 9th (6 days)
- 612 new transactions (2% of total in the data structure)
- 56MB -> 90MB in internal Motoko heap size growth.
I'll update this sheet periodically as more data comes in (weekly or something like that)
Update: Here's a graph of the last 18 days. Wasm memory remains flat and haven't seen any spikes causing wasm memory to increase during that time. GC looks to have kicked in once earlier this month (July 9th), memory increasing somewhat rapidly with relatively little usage. Makes me wonder how so much memory (~70MB) gets generated with just a few transactions.
Will send another update next month.