Some questions about working with cxx_* rules

[rickyelopez]

Hello, just getting started with buck2 so forgive me if some of these are obvious questions!

1. Is there any mechanism for providing a linkscript to cxx_binary other than adding the relevant flag+path to linker_flags? Adding the flag+path is fine, just wondering if I missed something
2. Is there an idiomatic way to produce assembly files when compiling with cxx_binary? I have accomplished this with a bit of starlark, but it seems... wrong. Also, related, is there a good way to investigate these types? I got the the starlark below by repeatedly printing out dir and type calls, which was super annoying.

       def _generate_asms_impl(ctx: AnalysisContext) -> list[Provider]:
           outputs = []
   
           for file, provider in ctx.attrs.binary[DefaultInfo].sub_targets["objects"][DefaultInfo].sub_targets.items():
               if file.endswith(".s.o") or file.endswith(".S.o"):
                   continue
   
               sub_targets = provider[DefaultInfo].sub_targets
               maybe_assembly = sub_targets.get("assembly")
               if maybe_assembly:
                   outputs.extend(maybe_assembly[DefaultInfo].default_outputs)
   
           return [DefaultInfo(default_outputs = outputs)]
   
       generate_asms = rule(
           impl = _generate_asms_impl,
           attrs = {
               "binary": attrs.dep(),
           },
       )
   
   

3. Same question as above but for other output targets, i.e. binaries, ihex, etc. I'm using cxx_genrule right now to call objcopy, which is fine too I guess.
4. Does the linker_extra_outputs parameter actually do anything? The docs say that adding files to it allows you to use the $(output ...) string expansion in the linker_flags list, but adding a file and trying to use that expansion results in this error: Can't expand unrecognized macros (output).
5. Is there a way to access a specific source file in an filegroup, for example in srcs in a call to cxx_binary?
6. Is there any way to read a yaml or json file in starlark? I've got some tools that produce a yaml output containing lists of source files and headers, and I would like to use that output in the parameters to a cxx_binary call.
7. Is there any way to force buck2 to rebuild certain targets without having to do a full buck2 clean? Alternatively, is there a way to do a "partial" clean of just specified targets? Thanks!

[zjturner]

Will do my best, even though someone on the buck2 team might be better equipped to answer some of them.

1. I don't think so. linker scripts are non portable and only work with specific linkers, but the rule is generic and is supposed to work with any linker. You could probably add a linker_script attr to the rule and have it be ignored (or error) for linkers that don't support it, but it would probably require some design agreement with the team on the api.

2. AFAIK, if you pass in a .S file into cxx_library() or cxx_binary(), it will automatically invoke the asm compiler that is configured in your toolchain. The cxx toolchain has two relevant providers, AsmCompilerInfo() and AsCompilerInfo(). I actually don't remember the difference, but I think one of them is effectively deprecated and doesn't do anything, and the other one is the real one. In any case, as long as you set the correct one and have it point to your assembler, you should just be able to pass asm srcs to a cxx_library / cxx_binary and it "just works".

For inspecting types, dir and type calls are indeed the best way to do it. You can also use buck2 audit providers to print out the entire provider tree of everything returned by a rule, but for large rules it can be noisy and difficult to parse..

3. Unsure about other output targets. e.g. I have no idea what ihex is. Is this some kind of baremetal file format that you have a custom toolchain which understands?

4. Easiest way to answer questions like this is to reverse engineer the logic in the prelude. A quick grep for it shows that several rules use it as an attribute, but its value is never read anywhere. So my guess is that it does nothing. If I had to guess, it's probably a holdover from buck1 that is there for rule api compatibility.

5. Unsure off the top of my head.

6. I don't think so, but someone correct me if I'm wrong. The closest thing I can think of that might work is to wrap the invocation of your compiler (defined by your toolchain) that cxx_binary invokes in a python script or something. Pass your yaml/json as an arg into the python script, along with the "real" args to the compiler. Also search the docs for dynamic dependencies, it's similar to what you're asking but not exactly the same.

7. Unfortunately no, and this is pretty annoying actually. A hacky workaround I've found is to add a define to the target you want to rebuild that doesn't do anything. This will force it to treat the target as "dirty" and rebuild all of its sources.

[zjturner]

Actually dynamic action looks like it works for number 6.

https://buck2.build/docs/api/build/AnalysisActions/#analysisactionsdynamic_output

The basic idea is that you can use dynamic_output to generate your yaml / json file, and then that will give you an object that has a read_json() method which returns a dict. From there you should be able to use that information to construct command line args to your tool.

[rickyelopez]

Hey @zjturner thanks for the response!!

1. I don't think so. [...]

Bummer, but no biggie. I will probably write my own rule that wraps cxx_binary to implement this.

2. AFAIK, if you pass in a .S file into cxx_library() or cxx_binary() [...]

Maybe I wasn't clear, what I meant to say that I wanted to produce the intermediate assembly output from compiling C files. This output is already added as a sub_target (as shown in the snippet I shared above) and I was just wondering if there was an intended way of actually getting the output to be produced. If not, the snippet I shared works fine, it just seemed strange to me that 90% of functionality would be built out but the last 10% would require some extra work like this.

3. Unsure about other output targets. e.g. I have no idea what ihex is. [...]

Yeah this is a baremetal target. ihex is a hex in intel format which contains some additional info, like the address that content belongs at. Many compilers/linkers that can target bare metal targets (such as, in this case, gcc-arm-none-eabi) can output this format.

4. Easiest way to answer questions like this is to reverse engineer the logic in the prelude. [...]

Yup this is what I did and I came to the same conclusion, I was just wondering if I had missed something. Seems strange for it to be well documented but then not be implemented at all. I guess I didn't!

6. Actually dynamic action looks like it works [...]

nice thanks for the tip, I'll check it out!

7. Unfortunately no [...]

Bummer, but not the end of the world.

Thanks for all the info!

[zjturner]

Maybe I wasn't clear, what I meant to say that I wanted to produce the intermediate assembly output from compiling C files. This output is already added as a sub_target (as shown in the snippet I shared above) and I was just wondering if there was an intended way of actually getting the output to be produced. If not, the snippet I shared works fine, it just seemed strange to me that 90% of functionality would be built out but the last 10% would require some extra work like this.

Ahh I see. Probably this isn't supported by the prelude rule. I like your subtargets approach, but maybe it could be added directly into the prelude cxx_library / cxx_binary rules. Then you just do:

cxx_library(name="foo", srcs=[...])

and then on command line you do buck2 build :foo[asm]. This seems generally useful, so I bet something along those lines would be a good candidate for upstreaming.

[zjturner]

BTW, you can't wrap a rule with another rule, but you can wrap a rule with a macro. This is probably what you meant anyway.

[rickyelopez]

and then on command line you do buck2 build :foo[asm].

Yes, that is exactly what I was looking for!

BTW, you can't wrap a rule with another rule, but you can wrap a rule with a macro.

Yep that's what I meant, thank you for the correction

[cormacrelf]

Re number 6

One big idea with Buck and Bazel is that the BUCK/BUILD files are an extra limited starlark dialect, so much so that you basically treat them as a data format. Instead of using YAML or JSON that's been generated to describe a build, you kinda want to adapt the tool to output BUCK files instead (or, as well). It's pretty simple to do, especially if you pipe the output through buildifier.

For a hacky fix, most JSON is a valid starlark expression, so you can just prefix it with contents = , save it as .bzl, now you can import it anywhere you like. Then make some macros that wrap cxx_binary and thread an imported JSON document into it. But this isn't very nice.

[zjturner]

Having a tool that runs at build time output another BUCK file won't really influence the current build though. Once a build starts, no changes to BUCK or bzl files get picked up until the build finishes and a new build is going to start.

[rickyelopez]

Once a build starts, no changes to BUCK or bzl files get picked up until the build finishes

Yeah this is what I was gonna ask. I had briefly considered trying this, but I had exactly this question and I guess that answers it.

[cormacrelf]

It's not so bad, just have a step in CI that enforces you keep the BUCK files up to date. A script that writes BUCK output + these two lines in CI is much simpler than trying to write a recursive dynamic dependencies thing and figure out how to call the CXX internals, and also deal with breaking changes to CXX build APIs (they are not versioned or documented, nor do they have any compatibility promise afaik) etc. It's a few days' work vs a month's if you're new to it.

./regen-buckfiles.sh
git diff --exit-code

Ideally when you do generate the BUCK files, they have file globs in them that mean you can rename files etc without regenerating. But not a big deal.

My repo went through an evolution that included this phase, with generated BUCK files. Ultimately I ended up writing the inverse, and generating the precursor from the BUCK files. This way the machine is doing backwards compatibility only, not an annoying step that slows down using the build system you use now (Buck). That involved:

• having hundreds of generated BUCK files to form a starting point
• a flag on a precursor file to switch off BUCK generation and lock in BUCK as the source of truth
• In my case, a wrapper macro for rust_library that generated a libname-Cargo.toml target as well, of a custom rule type
• a script to query these targets and then build all of them at the same time, and write the output to the source tree (as Cargo.toml files)