mimalloc icon indicating copy to clipboard operation
mimalloc copied to clipboard
verified publisher icon microsoft

Multiple debug print statements and hugepages mapped

Open TechnophobicLampshade opened this issue 2 years ago • 1 comments

Hi,

Info

I'm using mi-malloc 2.1.2, and starting my app with LD_PRELOAD=/path/to/libmimalloc.so with the following environment variables set:

MIMALLOC_ARENA_RESERVE: '100'
MIMALLOC_EAGER_COMMIT_DELAY: '1'
MIMALLOC_PAGE_RESET: '0'
MIMALLOC_PURGE_DELAY: '-1'
MIMALLOC_RESERVE_HUGE_OS_PAGES: '2'
MIMALLOC_SHOW_ERRORS: '1'
MIMALLOC_SHOW_STATS: '1'
MIMALLOC_VERBOSE: '1'
MIMALLOC_RESERVE_HUGE_OS_PAGES_AT: '0'

I see the following in stdout when my app starts:

mimalloc: process init: 0x7ff7c8af6740
mimalloc: secure level: 0
mimalloc: mem tracking: none
mimalloc: using 2 numa regions
mimalloc: numa node 0: reserved 2 GiB huge pages (of the 2 GiB requested)
mimalloc: option 'show_errors': 1
mimalloc: option 'show_stats': 1
mimalloc: option 'verbose': 1
mimalloc: option 'eager_commit': 1
mimalloc: option 'arena_eager_commit': 2
mimalloc: option 'purge_decommits': 1
mimalloc: option 'allow_large_os_pages': 0
mimalloc: option 'reserve_huge_os_pages': 2
mimalloc: option 'reserve_huge_os_pages_at': 0
mimalloc: option 'reserve_os_memory': 0
mimalloc: option 'deprecated_segment_cache': 0
mimalloc: option 'deprecated_page_reset': 0
mimalloc: option 'abandoned_page_purge': 0
mimalloc: option 'deprecated_segment_reset': 0
mimalloc: option 'eager_commit_delay': 1
mimalloc: option 'purge_delay': -1
mimalloc: option 'use_numa_nodes': 0
mimalloc: option 'limit_os_alloc': 0
mimalloc: option 'os_tag': 100
mimalloc: option 'max_errors': 16
mimalloc: option 'max_warnings': 16
mimalloc: option 'max_segment_reclaim': 8
mimalloc: option 'destroy_on_exit': 0
mimalloc: option 'arena_reserve': 1
mimalloc: option 'arena_purge_mult': 10
mimalloc: option 'purge_extend_delay': 1
mimalloc: option 'show_errors': 1
mimalloc: option 'show_stats': 1
mimalloc: option 'eager_commit': 1
mimalloc: option 'deprecated_eager_region_commit': 0
mimalloc: option 'deprecated_reset_decommits': 0
mimalloc: option 'large_os_pages': 0
mimalloc: option 'reserve_huge_os_pages': 2
mimalloc: option 'reserve_huge_os_pages_at': 0
mimalloc: option 'reserve_os_memory': 0
mimalloc: option 'deprecated_segment_cache': 0
mimalloc: option 'page_reset': 0
mimalloc: option 'abandoned_page_decommit': 0
mimalloc: option 'deprecated_segment_reset': 0
mimalloc: option 'eager_commit_delay': 1
mimalloc: option 'decommit_delay': 25
mimalloc: option 'use_numa_nodes': 0
mimalloc: option 'limit_os_alloc': 0
mimalloc: option 'os_tag': 100
mimalloc: option 'max_errors': 16
mimalloc: option 'max_warnings': 16
mimalloc: option 'max_segment_reclaim': 8
mimalloc: option 'allow_decommit': 1
mimalloc: option 'segment_decommit_delay': 500
mimalloc: option 'decommit_extend_delay': 2
mimalloc: process init: 0x7efd2e008dc0
mimalloc: secure level: 0
mimalloc: using 2 numa regions
mimalloc: numa node 0: reserved 2 GiB huge pages (of the 2 GiB requested)
mimalloc: process init: 0x7efd2e008dc0
mimalloc: secure level: 0
mimalloc: mem tracking: none
mimalloc: using 2 numa regions
mimalloc: numa node 0: reserved 2 GiB huge pages (of the 2 GiB requested)
mimalloc: option 'show_errors': 1
mimalloc: option 'show_stats': 1
mimalloc: option 'verbose': 1
mimalloc: option 'eager_commit': 1
mimalloc: option 'arena_eager_commit': 2
mimalloc: option 'purge_decommits': 1
mimalloc: option 'allow_large_os_pages': 0
mimalloc: option 'reserve_huge_os_pages': 2
mimalloc: option 'reserve_huge_os_pages_at': 0
mimalloc: option 'reserve_os_memory': 0
mimalloc: option 'deprecated_segment_cache': 0
mimalloc: option 'deprecated_page_reset': 0
mimalloc: option 'abandoned_page_purge': 0
mimalloc: option 'deprecated_segment_reset': 0
mimalloc: option 'eager_commit_delay': 1
mimalloc: option 'purge_delay': -1
mimalloc: option 'use_numa_nodes': 0
mimalloc: option 'limit_os_alloc': 0
mimalloc: option 'os_tag': 100
mimalloc: option 'max_errors': 16
mimalloc: option 'max_warnings': 16
mimalloc: option 'max_segment_reclaim': 8
mimalloc: option 'destroy_on_exit': 0
mimalloc: option 'arena_reserve': 1
mimalloc: option 'arena_purge_mult': 10
mimalloc: option 'purge_extend_delay': 1

and then, as it runs, a few further lines:

mimalloc: reserved 32768 KiB memory
mimalloc: reserved 32768 KiB memory
mimalloc: reserved 32768 KiB memory
mimalloc: reserved 32768 KiB memory
mimalloc: reserved 32768 KiB memory

In /proc/pid/smaps for my process, I see the following:

sudo cat /proc/2329696/smaps | grep Private_Hugetlb
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:       0 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:    2048 kB
Private_Hugetlb:       0 kB
Private_Hugetlb: 1048576 kB
Private_Hugetlb: 1048576 kB
Private_Hugetlb: 1048576 kB
Private_Hugetlb: 1048576 kB
Private_Hugetlb:       0 kB
... a lot more zeros

Zooming in on those 1GB maps:

203840000000-203880000000 rw-p 00000000 00:0f 255328531                  /anon_hugepage (deleted)
Size:            1048576 kB
KernelPageSize:     2048 kB
MMUPageSize:        2048 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
FilePmdMapped:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb: 1048576 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:    0
ProtectionKey:         0
VmFlags: rd wr mr mw me de ht sd
203880000000-2038c0000000 rw-p 00000000 00:0f 255328532                  /anon_hugepage (deleted)
Size:            1048576 kB
KernelPageSize:     2048 kB
MMUPageSize:        2048 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
FilePmdMapped:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb: 1048576 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:    0
ProtectionKey:         0
VmFlags: rd wr mr mw me de ht sd
20acc0000000-20ad00000000 rw-p 00000000 00:10 255328527                  /anon_hugepage (deleted)
Size:            1048576 kB
KernelPageSize:  1048576 kB
MMUPageSize:     1048576 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
FilePmdMapped:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb: 1048576 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:    0
ProtectionKey:         0
VmFlags: rd wr mr mw me de ht sd
20ad00000000-20ad40000000 rw-p 00000000 00:10 255328528                  /anon_hugepage (deleted)
Size:            1048576 kB
KernelPageSize:  1048576 kB
MMUPageSize:     1048576 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
FilePmdMapped:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb: 1048576 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:    0
ProtectionKey:         0
VmFlags: rd wr mr mw me de ht sd

two of them are using 1GB pages, two of them are using 2MB pages. Here are those 4 pages in /proc/mypid/numa_maps:

...
03e00000 bind:0 file=/anon_hugepage\040(deleted) huge anon=1 dirty=1 N0=1 kernelpagesize_kB=2048
04000000 bind:0 file=/anon_hugepage\040(deleted) huge anon=1 dirty=1 N0=1 kernelpagesize_kB=2048
04200000 bind:0 file=/anon_hugepage\040(deleted) huge anon=1 dirty=1 N0=1 kernelpagesize_kB=2048
04400000 bind:0 file=/anon_hugepage\040(deleted) huge anon=1 dirty=1 N0=1 kernelpagesize_kB=2048
2ce50000000 bind:0 anon=40960 dirty=40960 N0=40960 kernelpagesize_kB=4
203840000000 prefer:0 file=/anon_hugepage\040(deleted) huge anon=512 dirty=512 N0=512 kernelpagesize_kB=2048
203880000000 prefer:0 file=/anon_hugepage\040(deleted) huge anon=512 dirty=512 N0=512 kernelpagesize_kB=2048
20acc0000000 prefer:0 file=/anon_hugepage\040(deleted) huge anon=1 dirty=1 N0=1 kernelpagesize_kB=1048576
20ad00000000 prefer:0 file=/anon_hugepage\040(deleted) huge anon=1 dirty=1 N0=1 kernelpagesize_kB=1048576
7efd1c1fd000 bind:0
7efd1c1fe000 bind:0 anon=2048 dirty=2048 N0=2048 kernelpagesize_kB=4
...

Questions

  • My app, without mimalloc, does not make any 1GB mappings (it only makes a small number of 2MB mappings). I am confused by the extra 1GB-sized 2MB-pages mappings. are these expected?
  • Why do I see "numa node 0: reserved 2 GiB huge pages" logged three times in stdout (along with a lot of the settings rows that seem to be logged multiple times)?
  • These memory mappings are prefer:0 rather than bind:0. Is there a risk that these maps will fail to bind on node 0, which could be very bad for application performance?

Thanks!

TechnophobicLampshade avatar Dec 15 '23 18:12 TechnophobicLampshade

The other concerning detail that I just spotted is:

mimalloc: process init: 0x7ff7c8af6740
mimalloc: process init: 0x7efd2e008dc0
mimalloc: process init: 0x7efd2e008dc0

I'm only meant to be running a single process..

My app does have three threads though.

TechnophobicLampshade avatar Dec 15 '23 18:12 TechnophobicLampshade