buck2
buck2 copied to clipboard
facebook
Share dynamic values across dynamic_output actions
Adds support for sharing values across dynamic_output actions.
The function given to a dynamic output action can now return a list of providers. The ctx.actions.dynamic_output function returns a handle to the future values of these providers. Another dynamic output action can take these handles in, which allows the function to resolve these handles into their resolved values. This way it is possible to share values that are obtained within a dynamic output lambda with another dynamic output lambda.
Includes a small example demo
$ cd examples/dynamic_value
$ cargo run --bin buck2 build //:target_b
The motivation comes from Haskell builds, where we want to achieve cross package module granular builds as described in https://github.com/tweag/rules_haskell/issues/1631. In that case we need to know the module dependency graph across package boundaries and track a tset for each module that captures its transitive module dependencies across package boundaries. Without this feature this is not possible without piping these module dependency graphs through the filesystem and reloading and reanalyzing them over and over again in each packaeg's dynamic output action.
- Example using dynamic values
- Implement dynamic values
We have now successfully applied these changes on a Haskell project to achieve cross package module granular incremental builds. The dynamic value introduced here works to propagate module dependency graphs across package boundaries through tsets.
I'd like to open this up for review and discuss whether the API change is acceptable. This adds an example to test the feature. cc @cjhopman
Assigning the PR to myself.
However, this week we have internal conference, and next week I'm on holidays. Can it wait two weeks @aherrmann?
Hi @stepancheg, thank you for taking this on! Yes, of course, no problem if this waits two weeks. :slightly_smiling_face:
@stepancheg I hope you had a good conference and vacation! Did you have a chance to take a look at this since you're back?
I'm working on some changes in dynamic_output API which I plan to land before taking this PR. I'll rebase this PR on top of my work.
Just FYI, this PR is not forgotten.
@stepancheg Thanks for the update and for looking into it! Let me know if there's anything I can help with.
Hi @stepancheg, did you have a chance to look into this? Is there anything I can do to help this along?
@stepancheg has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.
Got distracted by buck2 starlark profiler needed to be fixed because many people reported the issue. This is still the top of my list. Again, sorry for delay.
Fyi, we're in the process of discussing/reviewing this internally at the moment. The big open question right now is whether you should have to declare whether a dynamic_outputs lambda expects or returns dynamic values, especially in the context of the new dynamic_actions APIs. Will keep things updated
@aherrmann we have finally merged your PR db40a075b624db1f67320a01a47ee1fc7624a1c8. Thank you, and sorry for delay.
API is not final, we plan to tweak it. Also this API is not implemented for BXL yet. But fundamentally it will stay as you proposed it.
@stepancheg That's fantastic news, thank you! I look forward to trying out the new API!
@stepancheg As a first test I took the example included in this PR and rebased it on the new API. I'm happy to report that it worked very well for that example, see https://github.com/facebook/buck2/pull/744.
closing, this was implemented in https://github.com/facebook/buck2/commit/db40a075b624db1f67320a01a47ee1fc7624a1c8