bazel-skylib icon indicating copy to clipboard operation
bazel-skylib copied to clipboard

Add helper rule/provider for working with files inside tree artifacts

Open alexeagle opened this issue 3 years ago • 3 comments

TreeArtifacts have a lot of shortcomings. One of them is that it's impossible for an action to produce labels both for a directory output (ctx.actions.declare_directory) and also a label for a file within that directory. (some parts of Bazel fail to strictly disallow this, but at least the RBE protocol fails when you do this.)

In rules_nodejs we are working on providing dependencies as directories. This is because in the npm ecosystem there's no equivalent of a .jar file, so our dependencies are exploded and easily reach 10k input files to an action, making sandbox setup time a significant portion of action execution time.

However there's no accepted way across the Bazel ecosystem to continue referencing one file within a directory. One use case where you need this in many languages is the "main". Take https://docs.bazel.build/versions/master/be/python.html#py_binary.main for example - if you wanted to supply a directory (TreeArtifact) to the srcs of a py_binary you'd have no way to say which file within that directory should be executed.

We'll probably invent our own solution just for rules_nodejs (https://github.com/bazelbuild/rules_nodejs/pull/2579) but we'd rather upstream this concept into skylib so that rulesets can be more interoperable when they need to pass each other a construct of type {Label of TreeArtifact, string path within that label}

alexeagle avatar May 25 '21 20:05 alexeagle

Update, we did invent a rules_nodejs-specific rule https://github.com/bazelbuild/rules_nodejs/blob/7f688a95c9002a9e5148f218dfa9f1ea6464f50a/nodejs/private/providers/directory_file_path_info.bzl but would still be better to share this, since it has a Provider symbol that ought to allow interop between rulesets.

alexeagle avatar Oct 09 '21 18:10 alexeagle

Update again, this is now in bazel-lib: https://docs.aspect.build/aspect-build/bazel-lib/v0.9.6/docs/directory_path-docgen.html

alexeagle avatar Apr 20 '22 20:04 alexeagle

Such a thing would be useful to standardize, whether in code or at least in documentation. @comius - what do you think?

tetromino avatar Jul 13 '22 19:07 tetromino