current_commit stat negative

[maxbachmann]

I am currently porting mimalloc to a new platform. When running it with my application I do get negative values for current_commit from mi_process_info after some time. It's a debug build so MI_STAT is set to 2. My port doesn't provide an implementation of _mi_prim_process_info so this is completely based on mimallocs own tracking. I am working off of the latest tagged version v2.1.7.

Is this expected? If not is there anything known that could cause this and that I could look into?

[daanx]

That looks like spam? -- I would not click on that link. Let me see if I can remove it.. (as an aside, I think the commit can be negative due to thread interaction but let me think about this a bit more and reply in more detail)

[maxbachmann]

From my understanding current_commit should match the amount of memory committed in prim.c. If I add my own atomic to track the calls in there the numbers match for a while and start to go out of sync once threads get destroyed.

These two numbers match for a while and then appear to go out of sync once threads start to get destroyed. At this point current_commit appears to get decreased by 32mb without any corresponding decommit in prim.c.

[maxbachmann]

@daanx you mentioned in the PR that you manually added the fix. Could you reference the corresponding commit here once it's in the public version of the repository?

[daanx]

Hi @maxbachmann -- ah, I am not sure what exact commit but in the latest dev / dev-slice it should be fixed I think; In particular, all OS statistics now use the _mi_stats_main and increment/decrement the same stats -- this prevents the issue where one thread commits, but another decommits (leading to negative commit in that thread). I also addressed over time the issue where a region would be decommitted but it already had partially decommitted pages in there. Let me know if the latest versions fix your issue?

[maxbachmann]

That explains why I couldn't find a specific commit. This only occured for me when I was permanently running mimalloc in the "uninitialized" mode (that's used before the static initializer is run). I have since finished up the playstation port and so this isn't the case anymore.

I can try whether I am still able to reproduce it by reverting to this broken behavior next week.

[daanx]

Great you have a playstation port ! :-) (I guess you cannot share the port 's prim.c ?).

In "uninitialized" mode there is a call at some point in process_load/init that resets the statistics to not "confuse" users with early allocations by the libc etc. That may also be something you observed?

[maxbachmann]

Yes unfortunately all system apis are under NDA :/ Sharing it with people that have signed the NDA (e.g. via the internal forums) would be possible.

I simply didn't call process_load in the initial version. Now we use the C++ Version using a static global. So we would only call the allocator while preloading if the allocation is in a different static global. That just doesn't happen in our application.

[achen-rbx]

@daanx Just to follow up on this thread, I am also trying to fetch current_commit from mi_process_info and it returns a negative number if MI_DEBUG is not set to a value greater than 0.

This was tested on an Android device and with v3.0.4.

If I set MI_DEBUG to a value greater than 0, then the current_commit generates a more reasonable number for committed memory.

I tried looking through the Unix implementation for decomitting and saw this block in unix/prim.c where needs_recommit is set to false for non-debug builds. Setting this to true gives the same reasonable number for the committed memory metric:

  #if !MI_DEBUG && MI_SECURE<=2
    *needs_recommit = false;
  #else
    *needs_recommit = true;
    mprotect(start, size, PROT_NONE);
  #endif

I was hoping to be able to fetch the current_commit on build that doesn't require MI_STAT 2 or MI_DEBUG; but it seems it isn't possible at the moment.