filprofiler icon indicating copy to clipboard operation
filprofiler copied to clipboard

Include mmap()ed segments and how much is paged in in report

Open itamarst opened this issue 5 years ago • 1 comments

And possibly even take that into account for peak calculations?

  • [ ] File backed
  • [x] Not file backed (#29)
  • [ ] Not file backed, but using /dev/zero (hopefully not too common)

itamarst avatar Jun 24 '20 19:06 itamarst

Some research notes:

  1. psutil can list memory mmaps, but doesn't do macOS anymore.
  2. For our use case of one's own mmaps, the reasons it removed it don't matter. https://jvns.ca/blog/2018/01/26/mac-memory-maps/ describes how to do it (in Rust).
  3. mincore will list which pages are resident.

Overriding mmap() seems... problematic? Or maybe not.

Issue 1: Reentrancy concerns

  1. We use it as malloc fallback on Linux during initialization; possibly can work around this (use a direct syscall instead?).
  2. Quite possibly it's used to e.g. load shared libraries. Maybe that's fine too though.

Issue 2: What does file-backed mmap mean for usage?

There are two kinds of mmaps-file backed and not-file-backed. File-backed you can't quite count the file as being resident, so would need to use some measure, but it might throw off the peak somewhat if we don't count it? Maybe peak check should check resident size for mmaped files, but that'll likely make it really slow.

(For not-file-backed there's the issue of overreserving, but I guess that's Not Fil's Problem, it'll just make the graph look a little wonky.)

Even if we get peak wrong, we can at least DTRT for OOM reporting.

Issue 3: Identifying anonymous mmaps

Private anonymous mmaps have two mechanisms; /dev/zero fd seems like it's not supported on macOS at least, which simplifies things. Hopefully no one uses it? But can support it eventually.

itamarst avatar Jun 24 '20 20:06 itamarst