Increased virtual memory use

[SimonSapin]

Switching to this crate with #[global_allocator] apparently caused Servo to consume 7x more virtual memory than with alloc_jemalloc: https://github.com/servo/servo/issues/19010. This seems surprising since the same jemalloc code is used ultimately, and its compilation options don’t seem to be significantly different. Or maybe I missed something?

[alexcrichton]

Nah yeah I'm not entirely sure I'd know what's going on :(

Maybe compilation options? Maybe a bug in the bindings? Unsure...

[arthurprs]

You should be able to print jemalloc stats for both builds, it might reveal something.

[mateon1]

How could I do that? The memory profiler in Servo only shows 3771.41 MiB -- vsize, with everything else under 250MiB.

Edit:

Full memory report

Begin memory reports
|
|  245.97 MiB -- explicit
|     192.53 MiB -- system-heap-unclassified
|      49.47 MiB -- jemalloc-heap-unclassified
|       3.96 MiB -- url(https://servo.org/)
|          2.19 MiB -- js
|             1.00 MiB -- gc-heap
|                0.68 MiB -- decommitted
|                0.27 MiB -- used
|                0.04 MiB -- unused
|                0.02 MiB -- admin
|             1.00 MiB -- non-heap
|             0.19 MiB -- malloc-heap
|          1.66 MiB -- layout-thread
|             1.39 MiB -- local-context
|             0.27 MiB -- stylist
|             0.00 MiB -- display-list
|          0.11 MiB -- dom-tree
|
|   54.34 MiB -- jemalloc-heap-active
|   51.24 MiB -- jemalloc-heap-allocated
|  139.00 MiB -- jemalloc-heap-mapped
|  188.79 MiB -- resident
|  192.72 MiB -- system-heap-allocated
| 3771.41 MiB -- vsize
|
End memory reports

[arthurprs]

Call malloc_stats_print with all 3 args set to null and it'll print something like this to stderr

For the stdlib one you probably need to declare it yourself as:

extern "C" {
    #[link_name = "je_malloc_stats_print"]
    pub fn malloc_stats_print(write_cb: extern "C" fn(*mut c_void, *const c_char),
                              cbopaque: *mut c_void,
                              opts: *const c_char);
}

[SimonSapin]

@mateon1 Affected versions of Servo use this crate, not the standard library’s alloc_jemalloc crate. Also, this function seems to be easiest to use without a callback but jemalloc_sys declares it not-null. So I’d recommend something like:

extern "C" {
    #[link_name = "_rjem_malloc_stats_print"]
    pub fn malloc_stats_print(write_cb: Option<extern "C" fn(*mut c_void, *const c_char)>,
                              cbopaque: *mut c_void,
                              opts: *const c_char);
}

unsafe {
    malloc_stats_print(None, 0 as _, 0 as _);
}

[mateon1]

In the last unaffected commit, it seems that neither _rjem_malloc_stats_print nor je_malloc_stats_print exist. I'll compile Servo without modification and look through the symbols.

EDIT: Seems like it's just je_stats_print, but there's also malloc_stats_print.

[mateon1]

stats from servo/servo@6f52b331b8c6c38405964cdeacbc9f771c70b18c [last good commit]: https://ipfs.io/ipfs/QmX4W1UegaFjrtyZV2hETgxhuGkYwAJ6vov5bBfcbRrDK6/stats.good.txt stats from servo/servo@959ce482dd9f2f8c469964b8c258bd3e45f7ca2b [first bad commit]: https://ipfs.io/ipfs/QmX4W1UegaFjrtyZV2hETgxhuGkYwAJ6vov5bBfcbRrDK6/stats.bad.txt

[arthurprs]

The settings look the same but there's something fishy with the number of calls to the allocator.

[gnzlbg]

What's the reason for this difference?

VMware, Inc.
softpipe
3.3 (Core Profile) Mesa 17.3.0-devel

[SimonSapin]

@gnzlbg That’s debugging information about the graphics rendering backend that is printed by Servo (or webrender?), it is unrelated to memory allocators.

[gnzlbg]

rust-lang/jemalloc was updated on 2017-10-09 to jemalloc 5.0.1, but jemallocator uses the version from 2017-07-26 which is 4.5.

So maybe switching from Rust's jemalloc to jemallocator on the 2017-10-18 downgraded servo's jemalloc version, resulting in the increased memory usage that you are seeing? That would only be possible if servo was being compiled with a nightly from after 2017-10-09 before the change to jemallocator was made. Was this the case?

[arthurprs]

The number of calls to the allocator is way off between the runs.

                            allocated      nmalloc      ndalloc    nrequests
small:                        5927368        36362         1438        35179
large:                        3510272          160          114          161
huge:                         3145728            1            0            1
total:                       12583368        36523         1552        35341

                            allocated      nmalloc      ndalloc    nrequests
small:                       16683632       105644         3910       140488
large:                       69279744          546          258          723
huge:                        18874368            5            0            5
total:                      104837744       106195         4168       141216