jucipp
jucipp copied to clipboard
cppit
High memory usage
... mostly for applications using gtk ... because of the really great idea to forbid inclusion of selected headers but instead require to include full gtk.h. This IMHO gets worse with gtkmm... For me this means that I can open 3-4 files in juci to fill half of my 4GB of RAM. Starting to edit those files adds some more MB. Compiling will then initiate swapping (and make my laptop unusable for some minutes)
I did the test with with three programs that use gtkmm, all show similar behaviour. I then tested with programs using Qt and I can open dozens of files to get a similar RAM usage.
Can juci do something about that? Or am I stuck here? I would have liked to start contributing, but not being able to use juci to code juci is somewhat... weird...
(Sidenote: YCM is even worse, for me it uses about 30% more memory. QtCreator (with clang backend) used 80% of RAM with the second file I opened. So juci already does fairly well :))
This is strange, on my OS X laptop opening src/source* and src/notebook.*, 8 files in total, uses around 1GB of memory (which is to be expected). I'll try on my Linux desktop when I'm at work in a few hours.
On Tue, 21 Jun 2016 00:11:52 -0700, Ole Christian Eidheim [email protected] wrote:
This is strange, on my OS X laptop opening
src/source*andsrc/notebook.*, 8 files in total, uses around 1GB of memory (which is to be expected). I'll try on my Linux desktop when I'm at work in a few hours.
It gets a tiny bit better if I run cmake with BUILDTYPE=Release. But still I am far off of your 1GB for 8 files... But maybe I am running into a glibc-thingy again. I already had issues with tmux where glibc didn't free memory as expected. I have to patch it to call malloc_trim(0)... If this should be the case here again I think the only solution might be to setup my Gentoo again and use something like musl/uclibc(-ng)...
What version of libclang are you using? I'll see if there are any differences there.
On Tue, 21 Jun 2016 01:15:35 -0700, Ole Christian Eidheim [email protected] wrote:
What version of libclang are you using? I'll see if there are any differences there.
I am running llvm-3.8.0. gcc is v4.9.3 (just in case the used STL/libstdc++ also matters).
Your llvm version is very new, but you gcc version is quite old! This should not affect the result though:)
Opening the same 8 files gave me these results on Debian Testing: libclang 3.6: 1.9GiB libclang 3.7: 1.9GiB libclang 3.8: 1.8GiB
Not sure why there is such a huge difference between OS X and Linux, but on OS X I use standard headers which does not include doxygen comments for instance, and of course, juCi++ is compiled with clang++ on OS X instead of g++ (on Linux). I tried turn off saving the doxygen comments in system headers, but it made no difference. I know binaries compiled with g++ is faster than when compiled with clang++, but I do not know if there are differences in memory consumption.
On Tue, 21 Jun 2016 02:38:03 -0700, Ole Christian Eidheim [email protected] wrote:
Your llvm version is very new, but you gcc version is quite old! This should not affect the result though:)
I know... I would like to switch to a newer gcc but with the abi_cxx-tag introduced in gcc >=5 it became incompatible with clang (link errors), which still is missing support for that feature (patches are available and under review since quite some time). I need clang for some packages (webkit-gtk, boost, inkscape, firefox, and some more) because (surprise) with gcc some files cause the gcc-process to aquire all available memory - and I just wont give it all I have ;)
Opening the same 8 files gave me these results on Debian Testing: libclang 3.6: 1.9GiB libclang 3.7: 1.9GiB libclang 3.8: 1.8GiB
Thx for testing, but still looks better than for me.
Not sure why there is such a huge difference between OS X and Linux, but on OS X I use standard headers which does not include doxygen comments for instance, and of course, juCi++ is compiled with clang++ on OS X instead of g++ (on Linux). I tried turn off saving the doxygen comments in system headers, but it made no difference. I know g++ is faster than clang++, but I do not know if there are differences in memory consumption.
As noted above for me in some cases clang uses (much) less memory than gcc, at least when compiling to binary and not only analysing for IDE-purposes.
I now ran juci with `MALLOC_MMAP_THRESHOLD_=1024 MALLOC_MMAP_MAX_=16777216" and parsing takes way longer ;) With 5 files opened (terminal.cc, window.cc, juci.cc, directories.cc, source.cc) it uses ~1.1GB which will end up at about the same memory consumption as you have (little bit less, probably caused by the actual files opened or (more likely) the aggressive trimming I do). So additionally to the in general higher memory consumption on linux my main issue might be that glibc is just bad at trimming...
I have no experience with those environment variables, but I forgot to mention that libclang parsing is slower on OS X as well. So one might have to decide then, low memory consumption or fast processing, not both!:)
Thank you yet again for looking into this. I'll check later those environment variables and try to understand it better.
On Tue, 21 Jun 2016 03:12:10 -0700, Ole Christian Eidheim [email protected] wrote:
Thank you yet again for looking into this. I'll check later those environment variables and try to understand it better.
I don't kow exactly how they play together, but I took them from here: https://bugs.python.org/issue11849
Have not tested this, don't have a multi-core linux pc available here, but I found some information on the environment variable MALLOC_ARENA_MAX which made some sense (http://man7.org/linux/man-pages/man3/mallopt.3.html). From what I understand so far: MALLOC_ARENA_MAX=1 will lead to less memory consumption but slower execution, and MALLOC_ARENA_MAX=[number of cores] will yield faster execution and more memory consumption.
This is very interesting, and a great way to control juCi++ and its memory/speed, and power consumption too I guess. High memory use lead to higher power consumption, so maybe setting MALLOC_ARENA_MAX=1 might improve the battery time? At least the battery life on OS X is quite ok when using juCi++, with its slow initial processing and low memory consumption. Improvement in battery life might be because, for the most part, only the initial libclang parsing is affected by the glibc settings, but the memory consumption stays (and affect battery life) when a high memory glibc setting is chosen. Feel free to correct me:)
Also saw that the git integration is not affecting the cpu and memory usage notably, so it should be safe to use on laptops:)
I just tested MALLOC_ARENA_MAX and it did not lead to reduced memory usage, however, running MALLOC_MMAP_THRESHOLD_=1024 MALLOC_MMAP_MAX_=16777216 juci worked pretty well, and reduced the memory usage by around 33% for me when opening all the src/source_* files in Debian Testing.