How to install bxl outputs to specific folder

[Overhatted]

I have the bxl function in the bottom of this post which I'm using to merge all the compile_commands.json in a repository, but the end result is in buck-out. Is there a way for me to move it to a specific location in the repository? For example something like: buck2 bxl create_compile_commands.bxl:create_compile_commands --out-dir .

I could, of course, just copy the file with the shell but the fact that the output folder has a hash in its name makes it kind of complicated. Also, here, it seems to be implied that in the future we could set a bxl function to auto-update its output when its inputs have changed. Not sure if that's true (it would be very useful) but the shell solution would also not work in that situation.

More specifically, clangd searches for a compilation database in any of the parent directories so I would like to have buck2 copy the merged compilation database into the root of the repository. I can also point it to a specific sub-directory of the repository but buck2 has the hashes in the folder path making this difficult to maintain.

If copying of bxl outputs to a specific directory is not a current or planned feature, how is the LSP server given the compilation database inside META?

load("@prelude//cxx:cxx_toolchain_types.bzl", "CxxToolchainInfo")

def _create_compile_commands(ctx):
    bxl_actions = ctx.bxl_actions(
        toolchains = "toolchains//:cxx",
        target_platform = "prelude//platforms:default",
    )

    all_targets = ctx.cquery().kind("cxx", "...")

    content = cmd_args()
    for target, value in ctx.build(map(lambda x: x.label.with_sub_target("compilation-database"), all_targets)).items():
        content.add(value.artifacts()[0])

    argfile = bxl_actions.actions.declare_output("comp_db.argsfile")
    bxl_actions.actions.write(argfile.as_output(), content)

    db = bxl_actions.actions.declare_output("compile_commands.json")
    toolchain_cxx = ctx.unconfigured_sub_targets("toolchains//:cxx")
    cmd = cmd_args(bxl_actions.toolchains[toolchain_cxx][CxxToolchainInfo].mk_comp_db[RunInfo])
    cmd.add("merge")
    cmd.add("--flatten")
    cmd.add(cmd_args(db.as_output(), format = "--output={}"))
    cmd.add(cmd_args(argfile, format = "@{}"))
    #cmd.hidden(entries.values())
    bxl_actions.actions.run(cmd, category = "cxx_compilation_database_merge")

    ctx.output.ensure(db)

create_compile_commands = bxl_main(
    impl = _create_compile_commands,
    cli_args = {},
)

[wendy728]

it seems to be implied that in the future we could set a bxl function to auto-update its output when its inputs have changed

This has not been implemented, though it's something we are still interested in for later on.

BXL always writes outputs to buck-out. The LSP workflows at Meta will often write the path of the compilation database artifact to stdout using BXL, then read the blob from stdout afterwards and do further processing. Here's an example of what you could do:

db = ctx.output.ensure(db)
ctx.output.print(db) # or db.abs_path() to get the absolute path to the project root

Or, if you would like some structured information in stdout, you could use print_json:

db = ctx.output.ensure(db)
outputs = {}
outputs.update({"my_db": db}) # or db.abs_path()
# add other bits of information you need
ctx.output.print_json(outputs)

Using ctx.output.print will get you the buck-out path into stdout, and you'll never need to deal with configuration hashes or the structure of the buck-out path, both of which are subject to change.

Here is some more information about ensured artifacts.

[zjturner]

I posted this elsewhere in another issue, but since this is relevant. Try this bxl. It should be quite a bit faster because it is based on aquery and doesn't need to merge (or even generate) any target-specific compilation databases since it works straight off the action graph. Curious to see how it compares in terms of speed for a large workload. For my use case the linked one is quite a bit faster.

[Overhatted]

@zjturner That is a very nice bxl, thanks for sharing. I think it's worth creating a PR and putting it into the prelude since most C++ users will need that. I left a comment in the gist noting one problem with the bxl.

I wasn't able to compare the performance with my implementation since it apparently stopped working with a recent buck2 upgrade. Although I'm sure it's much faster since it doesn't merge all those small files.

[zjturner]

I eventually realized I didn’t need to parse the args file to build the args, I could just have the compile_commands reference the argsfile as is. The majority of the time spent is in writing the json anyway. When I do it that way it took about 3.5 seconds for around 3500 files, down from about 25 seconds. so it’s way faster.

I think it’s worth modifying it also to write the compile_commands as an output artifact rather than to stdout, that way it can benefit from caching so that it won’t do any work if nothing has changed. But maybe that’s for another time. Next week Ill see about making a PR. I wonder where Id put it though, since there’s no “standard library” of bxl scripts in the prelude is there?

[Overhatted]

I would put it in https://github.com/facebook/buck2/tree/main/prelude/cxx/tools since there are already a few script there.

[Overhatted]

@zjturner Can you share that BXL under the MIT license or something similar? I would like to use it and make some improvements to it but I can't really do anything with it unless you properly share it.

[zjturner]

I’ll do it as soon as I’m in the office next week

[zjturner]

Ok, I made a PR. First time, there doesn't seem to be a way to explicitly add reviewers, so hopefully it just gets seen naturally. Not sure how the process works or if I need to explicitly tag someone.