Sarcasm
problems with large code base/templates
Hi, this is not exactly an issue, but a "comment" to share some thoughts and experience. First, It is very good project and very usefull one. Unfortunately libclang has a lot limitations .... I'm using irony mode+server, but back at the time I first downloaded it, it was problematic for me: Every opened file required ~200 mb of RAM, so if you open 20 files, just to look at code, this quickly took all the resources of the machine. So I did some mods to ease the problem(I haven't looked at libclang behaviour lately, to check whether these are still relevant). The most importat: when a new TU is created, I first do a .pch of it and save it. Then create the unit. When too many units are created - some predefined threshold of memory usage is reached, the least used TU(for a file) is "killed". Still malloc doesn't return the memory to the system, so malloc_trim needs to be called on some places. Now, creating a compilation unit takes about 7 seconds on my machine, so each time killed unit is revisited this wait happens. I do this thing with pchs because it saves me some seconds as well. A problem with my implementation is that when headers for a TU change, using TU gives strange results. Maybe .h files need to be monitored and when any of them changes, the TU should be reparsed. Unfortunately, I have no idea how to get the deps/(.h files recursively) for a TU by means of clang. Maybe I didn't look at the libclang methods extensively. Maybe a separate background thread should monitor and recompile at leas "active" TUs(the ones not "killed") or all TUs and reemit pch for inactive ones.
Another thing I did is to do the completion results filtering in the server. That is due to the fact, that in my project most of the time I get about 30 000 completions suggestions for each request. Sending that many results to irony.el is problematic, so I trim them and send only predefined number ~ 200.
Yet another proprosition I didn't implement(don't have skills in elisp) would be that each request to have an ID(auto incremented integer). This was related to a bug when multiple requests are sent - maybe this is fixed in irony el. By the way, there are issues with clang saving compilation units/translation units for completion - haven't tested wether they fixed it. If they did, this will be a good win for us.
Hope this is helpful.
Hi, this is not exactly an issue, but a "comment" to share some thoughts and experience. First, It is very good project and very usefull one. Unfortunately libclang has a lot limitations ....
I was wondering. Where your experience with libclang limitation from irony-mode usage or from a personal project you made using it or by using something else that irony-mode?
The most importat: when a new TU is created, I first do a .pch of it and save it. Then create the unit.
I don't see this coming anytime soon in irony-mode. I let libclang handle the pch generation for me, it handles update pretty well. I think the users can still do this on their own but I won't implement this kind of logic in irony-mode itself anytime soon.
For the memory usage, until now I haven't put much consideration into it but this is something I know I will have to do if I want to be serious. I think libclang provides some useful information and features to handle this, it is possible to serialize the translation units to disk (clang_saveTranslationUnit(), so I think the time you get of re-recreating the TU can be leveraged quite a lot with this feature. I would probably kill the least recently used translation units instead of targeting the one that consumes the most.
Another thing I did is to do the completion results filtering in the server. That is due to the fact, that in my project most of the time I get about 30 000 completions suggestions for each request. Sending that many results to irony.el is problematic, so I trim them and send only predefined number ~ 200.
I'm actually working on that, kind of! I'm trying to find a way to make the integration of irony-server commands more natural in Elisp, if I managed to get what I want this is the feature I want to implement first. Aside from parsing time, what takes a significant amount of time is to output the results, this feature should help significantly.
Yet another proprosition I didn't implement(don't have skills in elisp) would be that each request to have an ID(auto incremented integer). This was related to a bug when multiple requests are sent - maybe this is fixed in irony el.
That kind of think shouldn't be necessary.
By the way, there are issues with clang saving compilation units/translation units for completion - haven't tested wether they fixed it. If they did, this will be a good win for us.
Not sure what you are saying here, IIUC, you can only save a TU that compiles files so saving a TU with completion pending shouldn't work, and I don't think it make sense, you don't leave a file in the middle of a completion often. If it's a recently used file it will not be thrown out anyway. Maybe I didn't understand.
Hope this is helpful.
That's some interesting remarks, I wish I would reach this level of polishing and maintaining the level simplicity of irony-server, but I'm working at a very slow pace.
Hi, great project, I am running the irony mode now for a while and find it very useful. The only real issue I see is that over time the memory consumption goes up steadily, easily to 10G and more. Given the above statements, is that still the expected behaviour today?
Although my mac seems to handle this quite intelligently by swapping parts of it to disk, I'd rather have a cleaner solution like some memory threshold beyond which the oldest TUs are deleted (I understand from the above that the map storing these is driving the memory consumption).
