zzzprojects
Memory leaks when using IncludeOptimized and BulkSaveChanges together with PooledDbContextFactory [7.0.100]
I am using IncludeOptimize and BulkSaveChanges together with EFCore 7.0 in my project. I can see memory usage increase over time and figured out that this is caused by using "PooledDbContextFactory" instead of "DbContextFactory". It looks like some references are still being hold internally which are never released because the DbContext instance itself is never released but only resetted by the pool. It looks like this does not work for entities who are loaded by IncludeOptimized extensions.
Are there any known issues about that?
The job usually reads about 730 MB of entity data into memory, processes it, releases DbContext and sleeps for a while. Without normal DbContextFactory, memory usage does not increase. With PooledDbContextFactory, memory usage increases by about 60 to 80 MB on each run, that value seems to match to the size of all entities that are loaded by "IncludeOptimized".
Hello @schuettecarsten ,
The IncludeOptimized feature uses the QueryFuture to execute his queries.
We can see that a ConditionalWeakTable is used on this line: https://github.com/zzzprojects/EntityFramework-Plus/blob/master/src/shared/Z.EF.Plus.QueryFuture.Shared/QueryFutureManager.cs#L61C35-L61C139
So, if the context is never disposed of, the QueryFutureBatch related will never be disposed of either and will just continue to grow. That could probably explain the current behavior with the PooledDbContextFactory. However the Queries are cleared
once executed, so it doesn't totally explain why it grows by 60 to 80 MB every run.
Is it possible for you to try in a development environment and see if the same behavior happens with SaveChanges? If that's the case, that will give us a big hint that the troublemaker is probably the IncludeOptimized method.
Best Regards,
Jon
Hi @JonathanMagnan,
thank you for the explanation. I have done some tests and the memory leaks go away when I switch from PooledDbContextFactory to DbContextFactory. Using SaveChanges instead of BulkSaveChanges has no effect. I cannot use the standard Include because this will break the software due to excessive memory consumption.
Do not trust the 60 to 80 megabytes value, it's just what I could estimate from task manager while observing the software. I took a memory dump and analyzed it using Jetbrains dotMemory and could see lots of entities held by the DbContext instances that were stored in the pool.
Hello @schuettecarsten ,
Thank you for the info, so there is a huge chance that the memory leak is due to the ConditionalWeakTable as the DbContext is never really released.
We will continue to look at it, but you might have a few options you might want to try.
Clearing the cache
Using QueryFutureManager.CacheWeakFutureBatch.Clear(); to clear the ConditionalWeakTable (if you can) from time to time. To free some memory if we indeed have a memory leak on that feature.
https://github.com/zzzprojects/EntityFramework-Plus/blob/master/src/shared/Z.EF.Plus.QueryFuture.Shared/QueryFutureManager.cs#L61
Using IncludeGraph with the most recent version
It will not fix this issue and will require you to use method such as BulkMerge, but I find it worth mentioning as our new IncludeGraph dramatically decreases the memory usage: https://entityframework-extensions.net/v7-100-0-0-include-graph#memory-performance-improvements
If you test the Clear on the ConditionalWeakTable, let us know if that fixed the memory leak
@JonathanMagnan
During my tests I found an issue with BulkSaveChangesAsync, when trying to set IncludeGraph to true, I get the exception Oops! You cannot use the IncludeGraph option with LegacyIncludeGraph. Use one or the other but not both, but I did not touch LegacyIncludeGraph at all, only IncludeGraph to true. Setting LegacyIncludeGraph to false did not help.
@JonathanMagnan
Calling QueryFutureManager.CacheWeakFutureBatch.Clear() from time to time does not fix the memory issue, unfortunately.