server or clang preamble out of date after code refactoring with new header files?

[fspeech]

After I refactor some code into new header files they seem to get lost from the completer. Restarting Emacs restores the completer to a good state. I suspect restarting the server might as well. This may be a clang issue or it may be a known issue for Irony, in which case feel free to close this.

Regarding issue #131 GNU is taking their time releasing a Windows binary of Emacs 24.4. Until that is done I suspect Windows usage may be a problem for many users. OTOH my patched version of Irony is doing well for me with 24.3 so I am not in a hurry to make further progress on #131 either.

[Sarcasm]

Hi,

Did you save the header file?

In irony, right now I'm only sending the unsaved buffer content of the file being completed but to be thorough this should also be done with every header files directly or indirectly included. The server supports this theoretically but knowing which file are included is not necessarily trivial.

OTOH, when the file is saved, I expect libclang to be able to handle it.

Regarding #131, I'm checking for the GNU ftp regularly and nothing seems to appear, people seems to resort to build from individuals (http://semantic.supelec.fr/popineau/programming-emacs.html, https://bitbucket.org/Haroogan/emacs-for-windows/, ...). I don't like that very much so I understand your position.

Regarding #131, what is left is mostly to improve performances of big completions. Right now I'm more limited by irony-mode's functionality than its performances, so I'm working on the compilation database, issue #127 (and #130 that you submitted but was a duplicate). This mostly works for .clang_complete but I'm working on compile_commands.json support right now.

[fspeech]

Did you save the header file?

Yes I did save the headers. So it looks like libclang does not always rebuild preambles correctly? It is also possible that my system's time stamp is not working correctly (I use Msys2 and I noticed some funniness regarding time stamps). I will look into the time stamping problems on my system first.

[fspeech]

My system is okay but I am still having the problem. When first started everything works. Then somehow the server seems to lose track. Completer won't even prompt for things defined in the same buffer or prompt for std::string so I am guessing that something stopped libclang from compiling. Yet the command line clang++ compiled with no complaint. Irony log looks okay -- compiler options were correct. This also seems to be limited to some buffers, while other buffers are working correctly.

[fspeech]

Hmm sometimes the faulty buffers recover while others go faulty. I am wondering if my patched version of Irony (for issue #131) sometimes fell out of sync with the company mode front end. Working with a single buffer is fine. Once I started moving around buffers a lot problem starts to show. Anyway probably hold on until I get 24.4 to verify this is a problem with my patch instead of with Irony in general. It looks like that I may have to build Emacs from source when I get some time, if the official version continues to be held up.

[fspeech]

You mentioned https://bitbucket.org/Haroogan/emacs-for-windows/ . I may go with that one. Last time I checked it wasn't 24.4 but now it is.

[fspeech]

I am now using 24.4 and I seem to get the same problem. It is hard to pinpoint when it starts. It seems to work fine when I have one buffer and start having problems if I also open the header file, even if I save the header to disk after any modification.

[Sarcasm]

I just tested on Linux and I don't know how to reproduce your issue. Here are the steps I followed:

1. edit a .cpp file
2. edit the header file of a library included by the .cpp. I added a method to the library class.
3. save the header file
4. retrigger completion in the .cpp file. The new method is available in the candidate list. I guess this is where you have an issue but it seems to work for me.

You can edit the function irony--start-server-process to add debugging information to the log file.

1. M-x find-function RET irony--start-server-process RET
2. apply the following transformation:

             (start-process "Irony" irony--server-buffer
                            irony--server-executable
                            "-i"
+                           "-d"
                            "--log-file"
                            (expand-file-name
                             (format-time-string "irony.%Y-%m-%d_%Hh-%Mm-%Ss.log")
                             temporary-file-directory)))

[fspeech]

I turned on debug logging as you suggested. In the beginning there were no clang diagnostics (or just benign warnings) and I am getting normal completion suggestions. After a while (opening a header, loading another .cpp file, a bit of editing, and then going back to the very first .cpp file) I am getting diagnostic errors:

/proj/cpp/src/parser.hpp:1:2: error: unterminated conditional directive
...

and that is when the completion no longer works.

The diagnostic error does not make sense as it points to standard header guard:

#ifndef _PARSER_HPP_
#define _PARSER_HPP_

What I am wondering is this: both of my .cpp files refer to the same header file. Do they share the same preamble file? Is that managed by clang or irony? Could clang somehow get confused in the process?

Both .cpp files compile at command line with no problems.

[Sarcasm]

The diagnostic looks weird indeed if your header file is complete.

What I am wondering is this: both of my .cpp files refer to the same header file. Do they share the same preamble file?

I'm don't know exactly.

Is that managed by clang or irony?

Clang. Irony uses the default libclang options that have preamble enabled (CXTranslationUnit_PrecompiledPreamble).

Could clang somehow get confused in the process?

I'm don't know :S.

[fspeech]

I get the same problem when I just edit a single buffer in Emacs, but I changed the header using vim. So this definitely looks like for some reason libclang is not picking up updated header files correctly on my system.

[fspeech]

Restarting the server, which I guess restarts the libclang and triggers a new preamble to be generated, makes the problem go away.

[Sarcasm]

Hi,

Sorry for the delay. For me on Linux, whether or not the header is edited inside Emacs makes no difference and libclang is able to update the candidate list.

I have a new computer at home and Windows is not yet configured (no Emacs, no C++ compilers, CMake, ...). I can try it to see if I can reproduce it at least.

Ideally, if the issue is well understood and it comes from libclang, an issue should be made on their bug tracker.

Also it seems that the Emacs 24.4 binaries for Windows are now available: http://ftp.gnu.org/gnu/emacs/windows/?C=M;O=D

[fspeech]

I have been tolerating this bug for a while. It all points to somehow clang does not update precompiled headers at times. Is there a way to force a regeneration of the preamble from irony? Barring that another workaround is to have an irony restart command so that the server can be shut down and restarted with the same compiler options.

I can try to file a bug report but I think it would be difficult to report an UI issue there. A clean test case with C++ code to reproduce the bug will probably take some work.

[fspeech]

If I simply call irony--server-kill-process and then call irony--start-server-process I seem to get two sets of logs (one empty). So looks like that the start call would create a process but later another process is created again. So it would be sufficient to just bind the kill-process to a command? Or I should call irony--get-server-process-create which does setq irony--server-process and prevent a redundant creation of the server process?

[fspeech]

I vaguely recall that you plan to highlight errors returned by clang diagnostics? That would be very useful. Sometimes the completer doesn't return things as expected due to errors in the code.

[Sarcasm]

I can try to file a bug report but I think it would be difficult to report an UI issue there. A clean test case with C++ code to reproduce the bug will probably take some work.

Yup, I agree that it seems difficult to reproduce and I couldn't find a similar bug when looking for it in the LLVM/Clang bug tracker. There was some issues suggested regarding file locking and Windows I believe but I don't see how it would be related to your situation.

So looks like that the start call would create a process but later another process is created again. So it would be sufficient to just bind the kill-process to a command?

Yes, the process is created "on-demand". A sentinel is used to detect changes in the process status, so for example if you do M-x list-processes RET, you can kill the irony-server process manually.

I added the interactive command irony-server-kill to irony in commit 4554edf926697782539ad601bf20756a5c02a1f2, you can bind it to a keyboard shortcut now if you want to.

I vaguely recall that you plan to highlight errors returned by clang diagnostics? That would be very useful. Sometimes the completer doesn't return things as expected due to errors in the code.

I'm actually working on it and it looks like it's quite easy to implement. For now it will use a recent version of flycheck but I may add support for flymake in the future, since it's built-in Emacs.

I will bump this thread when a first draft is committed (I wish in the evening but things don't always go as planed).

[Sarcasm]

I'm not sure whether or not this is related but there is a recent post on the Clang Mailing List which describes some issues with Windows: http://lists.cs.uiuc.edu/pipermail/cfe-dev/2014-December/040254.html

[fspeech]

Thanks for the reference. Maybe it is related but it does not appear to be the same symptoms I am experiencing. But the group may be a good place to report the problem. Is your Windows system set up and working properly? It would be nice to get a confirmation before reporting a relatively intricate issue.

If I have been quiet lately it is because I've been using Irony productively and happily. There was a phase when I was actively changing my header files and this bug was a real nuisance. But now that my headers are mostly stable I have not encountered other significant issues and I am better and better at figuring out libclang's idiosyncrasies. Delay is also fairly acceptable even if sometimes I type faster than the completer can respond. Thanks for the great work! It really helped with the programming experience in C++!

[Sarcasm]

Is your Windows system set up and working properly? It would be nice to get a confirmation before reporting a relatively intricate issue.

I will try to do it in the following days and tell you.

I published a first version of flycheck-irony, you may want to give it a try (it may not be available on MELPA yet). This hasn't been tested much so feedback is always appreciated and I will try to use it at work starting tomorrow.

[Sarcasm]

Thanks for the reference. Maybe it is related but it does not appear to be the same symptoms I am experiencing. But the group may be a good place to report the problem. Is your Windows system set up and working properly? It would be nice to get a confirmation before reporting a relatively intricate issue.

I have finally setup the irony on Windows and I can replicate the issue, I have the following message:

c:/Users/papin_g/projects/irony/irony-mode-build/foo/foo.h:1:2: error: unterminated conditional directive

With flycheck this could help, sadly flycheck only print the errors in the current buffer and this error is in the header file so I think that will not help very much.

[fspeech]

I have finally setup the irony on Windows and I can replicate the issue, I have the following message:

Yep. That sounds like the exact same bug. There isn't anything wrong with the header file but the change threw off some internal caching mechanism of clang. So it is probably safe to report this as a bug to the clang devs. It is interesting that this rather glaring bug is not yet reported by anyone else.

[Sarcasm]

Yes, I agree with you, and I'm not sure the report will satisfy the Clang folks if we can't provide the steps to reproduce. But maybe they will be able to help us debug the issue.

[fspeech]

Now that you can reproduce this, if you don't mind, could you be the one to report it to the clang devs? Questions will arise with regard to what libclang functions are called and with what parameters and you are far more qualified to answer those questions than anyone else.

[Sarcasm]

Now that you can reproduce this, if you don't mind, could you be the one to report it to the clang devs?

Sure.

[Sarcasm]

Okay, so I made some progress, I still have to take some time to make a proper bug report for the Clang devs but I will dump my findings here in the meantime.

First let's assume a source file where completion occurs, there is no need to edit this file for the problem to appear:

foo.cpp:

#include "foo.h"

void Foo::

Then the header file that we are editing:

foo.h:

#ifndef FOO_H_
#define FOO_H_

struct Foo
{
  void foobar();
};

#endif /* !FOO_H_ */

Steps to follow:

1. Launch an interactive session of irony-server:

   ~/AppData/Roaming/.emacs.d/irony/bin/irony-server -d -i

2. Request code-completion. Please note that first one should enter the command, then the compile options, in this case there is no compile options so RET should be typed twice

   complete foo.h 3 11

3. Remove the method void foobar(); in the header file and trigger completion again.

4. Restore void foobar(); in the header file and trigger completion again. This should display something like the following:

   complete foo.cpp 3 11
   
   execute: Command{action=Command::Complete, file='foo.cpp', line=3, column=11, flags=[], unsavedFiles.count=0, opt=off}
   debug: complete: 2 diagnostic(s)
   ./foo.h:9:2: error: invalid preprocessing directive
   ./foo.h:1:2: error: unterminated conditional directive
   (
   ("Foo" 35 "" "" "Foo" 3 (""))
   ("foobar" 35 "void" "" "foobar()" 6 ("()"))
   ("operator=" 35 "Foo &" "" "operator=(const Foo &)" 9 ("(const Foo &)" 1 12))
   ("~Foo" 35 "void" "" "~Foo()" 4 ("()"))
   )
   
   ;;EOT
   

My hypothesis is that libclang internal buffers are getting corrupted somehow when editing the header files.

[fspeech]

Cool. It may be worthwhile to put this to the clang devs in case this is a known issue. While I don't doubt that they would prefer a clean test case designed to isolate the bug location, the effort to do that would be wasted if it is a known issue (at least to some devs there). Of course they likely don't know much about Irony so it would be nice if the report focus on how libclang is involved.

[Sarcasm]

Okay, so just for the record, I have made progress very slowly. I put this on the cfe-dev ML but without much feedback, and trying to reproduce the issue with a minimal libclang program doesn't seem to work. So maybe something is wrong in irony-mode usage of libclang, I will have to check the code.

[fspeech]

Thanks for the update. Does ycm use libclang the same way as irony? That may provide some clue?

[pfilipovv]

Yes, this preamble bug is really annoying. I confirm I came accross it too. Many times :)

[fspeech]

Greetings! I have not been actively working on C++ code for a while but I am now coming back to it. I have refreshed my Irony and clang install on MinGW and so far things appear to be working smoothly as far as completion is concerned. However I have not been heavily editing the header sections so this experience is limited. Is this preamble corruption issue still a real problem?