Calling rules_k8s outputs in another sh_binary does not work with guess_runfiles

[Globegitter]

We are trying to write a custom sh_binary that calls one of the k8s_object/k8s_objects output binaries but we explicitly have to set the env var PYTHON_RUNFILES=.. for it to work correctly. What is the point of guess_runfiles and is there a way to make this work out of the box for this use-case?

Example:

k8s_object(
  name = "deploy",
  ...
)

sh_binary(
    name = "my_script",
    srcs = ["my_script.sh"],
    data = [
        ":deploy.apply",
    ],
)

my_script.sh:

set -eux

bash -x path/from/workspaceroot/deploy.apply

We can see that the RUNFILES are pointing at a directory that does not exist. Adding export PYTHON_RUNFILES=.. to my_script.sh fixes the issue.

[nlopezgi]

iiuc, guess runfiles enables the run script to access all the files needed by the script as it is being executed in Bazel run (i.e., runfiles are typically set up to be accessed via bazel build, iiuc). I think a possible workaround would be to produce another output with a slight different command which would be consumed by downstream rules. The main reason why this is not supported, is that it breaks the bazel model to do these type of actions as part of bazel build. Specifically, kubectl actions are not hermeric nor deterministic wrt their inputs. Bazel only rebuilds a target when inputs change, and that behavior is very unlikely to properly support the current use cases for using kubectl. What is your use case (e.g what is the shell rule going to do) and how do you expect it to behave wrt when bazel decides to re build the output of the k8s_objects rule, and how do you expect to run the sh rule? If all you want is to wrap functionality of the rule around another rule (i.e. the sh_binary will just be used via bazel run to wrap k8s_object with more functionality), then I think the most viable solution will be indeed to have your shell script set the python runfiles en var to the correct value (note this is, in practice, what k8s_objects does to execute the outputs of several k8s_object actions, iirc).

[nlopezgi]

A bit more detail: in general, what we do not want, is for someone to be executing the output of these rules as part of a skylark rule that runs with bazel build. Wrapping them in a sh_binary and then running that via bazel run should be perfectly safe. In a way this is all that the objects.bzl rule does, if you see the template, all it does is set the RUNFILES env var (https://github.com/bazelbuild/rules_k8s/blob/master/k8s/resolve-all.sh.tpl#L24) and then run all of the commands (L26) - which is basically just pasting all the content from the executables produced for each object (https://github.com/bazelbuild/rules_k8s/blob/master/k8s/objects.bzl#L29). If your rule wants to do something similar to wrap around the executable of objects all it maybe needs to do is just include this line: (https://github.com/bazelbuild/rules_k8s/blob/master/k8s/resolve-all.sh.tpl#L24) and to be executed via bazel run which should set the env var?

[Globegitter]

To give some background, right now what we do for one of our projects is we build (in one target/rule) a backend for server-side rendering together with its frontend app and all its static assets, then we put this whole output of the build into a docker container and serve that. That worked well with the default rules as they ensured the container to be built and uploaded before applying the k8s manifest. But now we want to upload the static assets to a cdn, so we created a wrapper script which takes the assets of that build and uploads them and once that is successful calls k8s rules to ensure the docker container is uploaded and then applies the manifest.

Does that sound like a reasonable use to you? Then it might just be nice to add some documentation for this use-case (happy to provide a PR).

Edit: will also give it a try to add the guess_runfiles to our script but just adding the PYTHON_RUNFILES env var directly is also fine by me now that we know.

[ash2k]

@Globegitter what you want to do makes sense but it may not work, unfortunately. I'd like to do something similar (push multiple docker images using a single bazel run) but it doesn't work. I've described the issue I ran into here https://groups.google.com/d/msg/bazel-discuss/BoK0lojLWUU/Gu7ZRPv3AQAJ

[nlopezgi]

@ash2k, i responded in your thread, i do think your use case should be supported, iiuc, you just need to properly select the rules_go toolchain/platform https://github.com/bazelbuild/rules_go/pull/1762

[nlopezgi]

@Globegitter i do think your use case makes sense, as long as the executable is run in a bazel run command you should have no issues. Do let us know if my suggested solution works and it would be great if you could indeed help with docs pr once we're sure it works.