[Feature] Generate compilation database with compilation commands

[matthijskooijman]

trafficstars

It would be interesting if arduino-builder could generate a "compilation database", as a side effect of building a sketch. This is a JSON file containing the gcc commands used to compile each file in the project. Third party tools can use such a database to figure out the compilation flags used, to simplify analysis of the source code, or to easily support code completion in a third-party editor.

For docs, see http://clang.llvm.org/docs/JSONCompilationDatabase.html According to the YouCompleteMe docs, this file is customary called compilation_commands.json: https://github.com/Valloric/YouCompleteMe#option-1-use-a-compilation-database

Generating such a file is not entirely trivial, especially since files are currently compiled from a temporary directory (so the commands should probably be slightly faked using the original filenames) and because .ino files are preprocessed before being compiled (but for the purposes of completion, it might be sufficient if the flags for each .ino file can be read separatey).

This might need some further thought, but I wanted to create this issue to track any discussion around this topic.

[ghost]

this would be really nice but this project seems somewhat dead?

[matthijskooijman]

Which project?

[ghost]

arduino builder, last commit one year ago

edited: ah I see, got confused by that one: https://github.com/arduino/arduino-builder/commit/5fd49b6d4c14abc31ae9e1a9f18d4c1e7ba8b355

[IPlayZed]

Is this feature something which is being considered to be in development? SonarLint would require this as described here and this would integrate really well with the Arduino VSCode plugin.

[matthijskooijman]

arduino-builder is no longer developed, relevant code has been moved to arduino-cli instead, and that has a basic implementation of this feature implemented already. There's still stuff left for improvement, see e.g. https://github.com/arduino/arduino-cli/issues/849#issuecomment-770390848 for a (possibly) outdated list of some points.

[per1234]

Thanks @matthijskooijman. What do you suggest be done with this issue?

[matthijskooijman]

I was mostly ignoring it because arduino-builder isn't really relevant anymore, but I guess arduino-builder is actually still used in the java IDE to run the build code from arduino-cli, right? I wonder if that means that this compilation database is actually generated by the Java IDE now as well?

In any case, I guess we can close as wontfix, since any usecases that need the compilation database, like the vscode plugin suggested above, are likely going to be using arduino-cli anyway?

[per1234]

I was mostly ignoring it because arduino-builder isn't really relevant anymore, but I guess arduino-builder is actually still used in the java IDE to run the build code from arduino-cli, right?

That is correct, so probably not quite ready to be considered irrelevant yet since it is very much in use by many people.

It is certainly not under active development, but on the other hand it doesn't need much development because at this point it is pretty much just a wrapper around Arduino CLI used to translate the arduino-builder interface.

I wonder if that means that this compilation database is actually generated by the Java IDE now as well?

It is not:

$ arduino_debug --version
Loading configuration...
Initializing packages...
Preparing boards...
Arduino: 1.8.19

$ BUILD_PATH="/tmp/arduino-build-path"

$ mkdir "$BUILD_PATH"

$ arduino_debug --verify --board arduino:avr:uno --pref build.path="$BUILD_PATH" Foo.ino
Loading configuration...
Initializing packages...
Preparing boards...
Verifying...
Sketch uses 444 bytes (1%) of program storage space. Maximum is 32256 bytes.
Global variables use 9 bytes (0%) of dynamic memory, leaving 2039 bytes for local variables. Maximum is 2048 bytes.

$ ls -1 "$BUILD_PATH"
Foo.ino.eep
Foo.ino.elf
Foo.ino.hex
Foo.ino.with_bootloader.bin
Foo.ino.with_bootloader.hex
build.options.json
core/
includes.cache
libraries/
preproc/
sketch/

The reason is because arduino-builder is missing this: https://github.com/arduino/arduino-cli/blob/60c1c98ef90cab5fb5c278c2a33e4e4edde85fd7/commands/compile/compile.go#L153-L155

	builderCtx.CompilationDatabase = bldr.NewCompilationDatabase(
		builderCtx.BuildPath.Join("compile_commands.json"),
	)

The required change to arduino-builder:

--- a/main.go
+++ b/main.go
@@ -45,6 +45,7 @@ import (
        "syscall"

        "github.com/arduino/arduino-builder/grpc"
+       bldr "github.com/arduino/arduino-cli/arduino/builder"
        "github.com/arduino/arduino-cli/arduino/cores"
        "github.com/arduino/arduino-cli/legacy/builder"
        "github.com/arduino/arduino-cli/legacy/builder/i18n"
@@ -258,6 +259,10 @@ func main() {
                ctx.BuildPath, _ = buildPath.Abs()
        }

+       ctx.CompilationDatabase = bldr.NewCompilationDatabase(
+               ctx.BuildPath.Join("compile_commands.json"),
+       )
+
        // FLAG_BUILD_CACHE
        if *buildCachePathFlag != "" {
                buildCachePathUnquoted, err := unquote(*buildCachePathFlag)

So it seems that this issue is still not resolved. It is also arduino-builder specific and thus not appropriate for transfer to the Arduino CLI repo.

I don't feel knowledgeable enough about the subject matter to determine whether or not it should be considered "wontfix". I'll leave that decision up to you or one of the other maintainers.