Follow symlink outputs (maybe opt-in)

[burdiyan]

I know that I'm doing something that's against the rules of Please. But I've been doing the same thing in Bazel and it handles this use case well, so maybe Please could also do this.

The idea is to allow build_rule to produce outputs that are symlinks to files generated in the source tree, and make Please follow the symlink to determine if the rule is dirty.

Right now Please won't rerun the rule if the file that symlink points to changes or disappears. And Bazel does this.

Bazel still warns you if you're using genrule for that, but if you're doing it via a custom rule - it works. Would be great if Please could do the same.

[Tatskaari]

I'm not sure I understand fully but I think we have something similar. We have plz generate that will symlink generated files from plz-out/gen into the source tree so IDEs and such work a bit better. There's also --update_gitignore= that can be used to add these generated files to your .gitignore to exclude them from source control. Does that satisfy the same needs?

[burdiyan]

I’ve just seen this in the docs. Looks like this was added after the last time I’ve looked into Please. Will get more info on that.

But my use case is actually the other way around: I want to store symlink for files in the source tree inside plz-out. This is totally possible now in Please, but it will only check the existence of the symlink and won’t follow it to check the contents.

[burdiyan]

Speaking of Bazel, this is how you can declare a symlink output in an action: https://docs.bazel.build/versions/main/skylark/lib/actions.html#declare_symlink

[Tatskaari]

Interesting... Could you tell me your specific use case so I understand what you're trying to achieve? Where are these symlinks going from and to? If they're going from plz-out/gen to the source tree, can't you just use a filegroup to add them as an output of a build rule?

Typically, Please projects don't check in their generated sources, but when they do, we just add them to rules like you would any other source. Sorry if I'm being slow.

[Tatskaari]

Speaking of Bazel, this is how you can declare a symlink output in an action: https://docs.bazel.build/versions/main/skylark/lib/actions.html#declare_symlink

Oh wait, if I understand correctly, the actions are similar to the cmd = "" on a build_rule(). If you wanted to output a symlink, you could ln -s and declare it in the outs = [] list. Are you just asking if we can make Please handle that better by following the symlink and hashing it's target rather than the symlink itself?

[burdiyan]

Oh wait, if I understand correctly, the actions are similar to the cmd = "" on a build_rule(). If you wanted to output a symlink, you could ln -s and declare it in the outs = [] list. Are you just asking if we can make Please handle that better by following the symlink and hashing it's target rather than the symlink itself?

@Tatskaari right, exactly that! I guess consequentially, this should only be allowed if local = True so that the output doesn't get cached, and rule doesn't get executed remotely.

[burdiyan]

The use case is a bit beyond the goals of Please, because it can become a big foot gun in terms of hermeticity and reproducibility. But if you know what you're doing you'll be fine. The idea is to provide more convenience trading off some reproducibility/hermeticity/correctness.

For example, this could be used to run yarn install whenever yarn.lock changes.

One could create rule like that:

build_rule(
    name = "yarn",
    srcs = ["package.json", "yarn.lock"],
    outs = ["node_modules/.yarn-integrity"], # This file is created by Yarn
    cmd = """
# Escape from the $TMP_DIR and go to the source tree. The actual process doesn't matter.
# For example "cd $(git rev-parse --show-toplevel)".
yarn install
ln -s node_modules/.yarn-install $OUT
"""
) 

You can still do this now, but if you remove .yarn-integrity file Please won't detect that it needs to rerun the rule. But if one can instruct Please to follow symlinks on a per-rule basis, this could help.

This trick could let people use Please as a general-purpose task orchestration tool, similar to make. And this can become very powerful, but very dangerous if you don't know what you're doing.

[burdiyan]

Update:

I've been playing around with Bazel a bit, and it actually follows the symlink by default. If you want to create a symlink and treat it as a symlink without dereferencing (which is what Please does by default) you have to you ctx.declare_symlink action in the Bazel rule.

Would it be hard to change this behavior in Please or make symlink dereferencing opt-in?

[burdiyan]

I've seen that there's been many GitHub issues created over time from people requesting different ways to turn the "convenience/reproducibility" gauge more to the convenience side. Seems like it's common for simpler projects to trade off some of the reproducibility and correctness for more convenience. I think this symlink trick gives enough flexibility for not very ugly workarounds, and probably is the easiest to implement in code than any other solution I could think of.

[Tatskaari]

Sorry for the slow reply, I was sick that week and missed your responses. I think this generally makes sense but requires some careful thinking. I'll add it to the backlog for v17 (we're gearing up to start work on that as we speak).

[stale[bot]]

This issue has been automatically marked as stale because it has not had any recent activity in the past 90 days. It will be closed if no further activity occurs. If you require additional support, please reply to this message. Thank you for your contributions.