File location changes in an output directory are not picked up

[lvanengelen]

Hi,

We have a build where there are rules like the following:

genrule(
    name = "ruleA",
    srcs = [
        # ...
    ],
    outs = {
        "lib": ["lib"],
        # ...
    },
    cmd = """
    # Do build things including building and running some code that downloads some libraries to `lib`.
    """
)

genrule(
    name = "ruleB",
    srcs = {
        "lib": [":ruleA|lib"]
    },
    out = "lib-b",
    cmd = """
      # copy stuff from `lib` to `lib-b`
    """
)

We changed the sources of ruleA to download the libraries to some other subdirectory of lib, but the libraries themselves did not change. To our surprise we then found that building ruleB took an older output from the cache, while the files were correctly downloaded in ruleA.

Diving into this further I came up with the following test setup:

genrule(
    name = "a",
    srcs = ["dirspec.txt"],
    outs = {
        "out1": ["destination.txt"],
        "out2": ["base"]
    },
    cmd = """
    d="$OUTS_OUT2"
    # Read location from source
    d="$d/$(cat "$SRC")"
    # Put the location in $OUTS_OUT1, this is just for verification.
    echo "$d" > "$OUTS_OUT1"

    # `greeting.txt` always contains the same text,
    # but its _location_ depends on the _contents_ of sources.
    mkdir -p "$d"
    echo "Hello, World!" > "$d/greeting.txt"
    """
)

With this setup I try to capture the idea of having a source file (dirspec.txt) determining the location of an output file (greeting.txt) inside an output directory (base) which I think corresponds with the above situation.

When I build :a with dirspec.txt containing dir1 the output directory will contain base/dir1/greeting.txt. When I then change dirspec.txt to contain dir2 and re-build :a I see that the command is run but the output directory still contains base/dir1/greeting.txt instead of the expected base/dir2/greeting.txt. The destination.txt file, however, is updated to contain the new path base/dir2 as expected.

Adding echo "$d" > "$d/changing.txt" to the rule, makes sure that there is a changed file in base whenever the location is changed. Re-building :a then ends up with the expected result.

Now I have the following questions:

• Is the above expected behavior?
• It looks like Please takes the data content of base into account, but not the paths. If it takes the effort of taking the file contents into account, then why not the paths of those files?
• It seems that using a directory in outs is dangerous if a change can result in files being moved without being changed. Is there any benefit of supporting a directory in outs?
• I learned about output_dirs = ["_outs/**"], which seems to almost do what we want, but in a rule taking the result as srcs, $SRCS is set to something like a/a.txt b/b.txt b/c/bc.txt requiring looping over all files to create the correct output directories. Our current alternative is passing around archives created using arcat instead of directories. Is there a more idiomatic way handle our scenario?