failed to handle @file directive in compile_commands.json

[jamesruan]

Please describe the problem.

I have compile command like following working

{
  "directory": "/Users/rbh/inlight-board/apps/canvas/out/ecc_wasm_pthread",
  "command": "/Users/rbh/emsdk/upstream/emscripten/em++ --config=CMakeFiles/canvas.dir/includes_CXX.rsp  -fno-rtti -Wthread-safety -Wall -Werror -Wextra -Wno-error=deprecated-declarations -Wno-error=implicit-int-float-conversion -Wno-unused-parameter -Wno-error=shorten-64-to-32 -Wno-missing-field-initializers -Wno-deprecated-literal-operator -g -fvisibility=hidden -fvisibility-inlines-hidden -Xclang -fno-pch-timestamp -fwasm-exceptions -pthread -Og -g -std=gnu++20 -o CMakeFiles/canvas.dir/version.cpp.o -c /Users/rbh/inlight-board/apps/canvas/out/ecc_wasm_pthread/version.cpp",
  "file": "/Users/rbh/inlight-board/apps/canvas/out/ecc_wasm_pthread/version.cpp"
}

However the following failed

{
  "directory": "/Users/rbh/inlight-board/apps/canvas/out/ecc_wasm_pthread",
  "command": "/Users/rbh/emsdk/upstream/emscripten/em++  @CMakeFiles/canvas.dir/includes_CXX.rsp  -fno-rtti -Wthread-safety -Wall -Werror -Wextra -Wno-error=deprecated-declarations -Wno-error=implicit-int-float-conversion -Wno-unused-parameter -Wno-error=shorten-64-to-32 -Wno-missing-field-initializers -Wno-deprecated-literal-operator -g -fvisibility=hidden -fvisibility-inlines-hidden -Xclang -fno-pch-timestamp -fwasm-exceptions -pthread -Og -g -std=gnu++20 -o CMakeFiles/canvas.dir/version.cpp.o -c /Users/rbh/inlight-board/apps/canvas/out/ecc_wasm_pthread/version.cpp",
  "file": "/Users/rbh/inlight-board/apps/canvas/out/ecc_wasm_pthread/version.cpp"
},

Logs

with @file version, nothing found in the command:

I[23:24:58.890] Loaded compilation database from /Users/rbh/inlight-board/apps/canvas/compile_commands.json
I[23:24:58.890] ASTWorker building file /Users/rbh/inlight-board/apps/canvas/src/canvas/path/renderer-proxy.cc version 0 with command
[/Users/rbh/inlight-board/apps/canvas/out/ecc_wasm_pthread]
/Users/rbh/emsdk/upstream/emscripten/em++ --driver-mode=g++ -fno-rtti -Wthread-safety -Wall -Werror -Wextra -Wno-error=deprecated-declarations -Wno-error=implicit-int-float-conversion -Wno-unused-parameter -Wn\
o-error=shorten-64-to-32 -Wno-missing-field-initializers -Wno-deprecated-literal-operator -g -fvisibility=hidden -fvisibility-inlines-hidden -Xclang -fno-pch-timestamp -fwasm-exceptions -pthread -Og -g -c -std=g\
nu++20 -resource-dir=/opt/homebrew/Cellar/llvm/17.0.6/lib/clang/17 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -- /Users/rbh/inlight-board/apps/canvas\
/src/canvas/path/renderer-proxy.cc

with @file changed to --config, It normal:

I[23:31:59.311] Loaded compilation database from /Users/rbh/inlight-board/apps/canvas/compile_commands.json
I[23:31:59.311] ASTWorker building file /Users/rbh/inlight-board/apps/canvas/src/canvas/path/renderer-proxy.cc version 0 with command
[/Users/rbh/inlight-board/apps/canvas/out/ecc_wasm_pthread]
/Users/rbh/emsdk/upstream/emscripten/em++ --driver-mode=g++ --config=CMakeFiles/canvas.dir/includes_CXX.rsp -fno-rtti -Wthread-safety -Wall -Werror -Wextra -Wno-error=deprecated-declarations -Wno-error=implici\
t-int-float-conversion -Wno-unused-parameter -Wno-error=shorten-64-to-32 -Wno-missing-field-initializers -Wno-deprecated-literal-operator -g -fvisibility=hidden -fvisibility-inlines-hidden -Xclang -fno-pch-times\
tamp -fwasm-exceptions -pthread -Og -g -std=gnu++20 -o CMakeFiles/canvas.dir/src/canvas/path/renderer-proxy.cc.o -c -resource-dir=/opt/homebrew/Cellar/llvm/17.0.6/lib/clang/17 -isysroot /Applications/Xcode.app/C\
ontents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -- /Users/rbh/inlight-board/apps/canvas/src/canvas/path/renderer-proxy.cc

System information

Output of clangd --version:

Homebrew clangd version 17.0.6 Features: mac+xpc Platform: arm64-apple-darwin22.5.0

Editor/LSP plugin:

Emacs+lsp-mode

Operating system:

Mac

[HighCommander4]

See also https://github.com/llvm/llvm-project/issues/69690 for another report of what looks like potentially the same regression from clangd 16 to clangd 17.

[sonictk]

Yea, pretty sure that this is what we're seeing as well and that this is a regression. We make heavy use of response files in Unreal here because otherwise the compilation commands are just too long on Windows and on Clang 16.0.6 the exact same compile_commands.json works fine but upgrading to anything in 17.x.x immediately breaks the compilation due to generating wrong compilation commands. When inspecting what the LSP generates for the MSVC compiler it's clear that all the information from the response files are being omitted.

[recursivenomad]

Checking in to say that GCC @/path/to/file response files work again in clangd 18.1.3, but only from within compile_commands.json. I am unable to get response files to work (at least on Windows) when added to my config.yaml, like below:

CompileFlags:
  Add: [ "@C:\\absolute\\path\\to\\response-file.txt" ]

Note that when inserting the contents of the quoted flag directly into each of my compile_commands.json commands, it works as expected; so I am assuming the syntax is appropriate, and response file are not being parsed in clangd configs.

[HighCommander4]

I am unable to get response files to work (at least on Windows) when added to my config.yaml, like below:

CompileFlags:
  Add: [ "@C:\\absolute\\path\\to\\response-file.txt" ]

Note that when inserting the contents of the quoted flag directly into each of my compile_commands.json commands, it works as expected; so I am assuming the syntax is appropriate, and response file are not being parsed in clangd configs.

The reason for this is that when processing compile commands, clangd expands response files (meaning, replaces the reference to the response file with its contents) before applying edits coming from CompileFlags: (Add: or Remove:).

This order of operations is actually deliberate. Response files are typically generated by the build system, and reflect the flags passed to the build compiler. Edits are often needed to remove flags to the build compiler that clangd doesn't recognize e.g. Remove: [-fblahblah]. If edits were applied before expanding response files, the Remove: would not catch flags that were found inside the response file.

[recursivenomad]

I definitely agree that the order of operations to apply CompileFlags: after compile_commands.json makes sense, but is there any reason not to also parse response files found in CompileFlags: once clangd gets there?

[HighCommander4]

is there any reason not to also parse response files found in CompileFlags: once clangd gets there?

No, I think that would be a reasonable enhancement.

[recursivenomad]

Cool 🙂 Would you like me to open a separate feature request for that, or is this thread sufficient?

[HighCommander4]

Cool 🙂 Would you like me to open a separate feature request for that, or is this thread sufficient?

I went ahead and filed one at https://github.com/clangd/clangd/issues/2079.

For this issue, since its title mentions compile_commands.json, let's close it as a duplicate of https://github.com/llvm/llvm-project/issues/69690.

[HighCommander4]

Duplicate of https://github.com/llvm/llvm-project/issues/69690