WIP: Include caching improved

[matthijskooijman]

trafficstars

This PR fixes an issue with the include detection caching where it would not use cached versions because of errors during compilation. This changes the include caching code to generate the same .d files as the compilation process, so they can be used even when compilation fails (which is convenient when working through multiple compilation errors in a big project).

The last commit in this PR is the core of it and is still a work-in-progress. It currently works, but it re-uses the same recipe.preproc.macros that must be modified to take advantage of this new code. However, this recipe is also used by the "preprocess the sketch" code during prototype insertion which currently also overwrites the .d file (which is probably not ideal, perhaps this needs to be changed to pass /dev/null instead?). An alternative approach would be to add a new recipe for include detection and leave the recipe.preproc.macros for just generating a preprocessed version of a file, without any .d files being generated (and adding a new recipe that generates a .d file and errors, but not any output, perhaps using -MM instead of -MMD to send .d file output to the file specified with -o).

[ArduinoBot]

:white_check_mark: Build completed.

:arrow_down: Build URL: http://downloads.arduino.cc/PR/arduino-builder/arduino-builder-217.zip

:information_source: To test this build:

1. Replace arduino-builder binary (you can find it where you installed the IDE) with the provided one

[facchinm]

I can see a huge improvement while developing big projects, but the extra complexity seems a bit overwhelming for smaller ones :slightly_smiling_face: I remember that @ffissore had some funny times trying to find out a common dependency method which could work even for older compilers. If -MF and -MT presence is totally optional I'd merge this PR and leave the choice whether or not supporting them to the core developers.

[ArduinoBot]

:white_check_mark: Build completed.

:arrow_down: Build URL: http://downloads.arduino.cc/PR/arduino-builder/arduino-builder-217.zip

:information_source: To test this build:

1. Replace arduino-builder binary (you can find it where you installed the IDE) with the provided one

[matthijskooijman]

It seems reusing recipe.preproc.macros is not a good idea. I just rebased on top of #236, and now:

• If you add the needed -MMD, -MT and -MF options, gcc errors out with cc1plus: error: to generate dependencies you must specify either -M or -MM. This is because arduino-builder now more effectively filters out the -MMD option.
• If you would then not filter out the -MMD option, this would work. However, for platforms that do not specify an explicit depfile name using -MF, would again run into problems with gcc trying to create /dev/null.d.

Hence, a new recipe is probably a good idea. Not sure if -o /dev/null -MMD -MF {dep_file}, or -MM - {depfile} should be preferred yet.

Not that I have not pushed the rebased version yet, since it would completely clutter this PR with all of the commits of #236, so I'll push once that one gets merged.

[CLAassistant]

All committers have signed the CLA.