Linter ignores part of the tree

[afeinman-snap]

We have a rather large protobuf repository that's being verified with buf lint during the premerge step. We've noticed that at least in one of the subdirectories the errors are being ignored. If I try to run build command, no proto from that subtree makes it into the generated json. And yet the buf.yaml we have does not mention this subdirectory anywhere at all.

1. Are there means to make buf print the name of the files it is looking at?
2. What would be the suggested troubleshooting procedure.

Unfortunately, since this is a proprietary commercial codebase, I cannot share any part of it including the configuration file. This behavior exists both in buf 1.4.0 and 1.7.0

[bufdev]

We're on Buf 1.14.0 now so I'd recommend upgrading. You can list the files with "buf ls-files". We're happy to help, but we would need a specific reproduction to be able to assist.

[afeinman-snap]

I've tested with 1.14 - same behavior. The ls-files command also skips the folder in question. Any chance you could point me to the place in the code where the tree is traversed? I will add some debug prints and retest

[afeinman-snap]

Alright. I guess I know what's happening. One of the excluded directories has soft link to the one that is missing from enumeration. Consider the following tree:

proto |
      |- a |
      |    |- b -> ../b
      |
      |- b |
           |- test.proto

If my buf.yaml has:

build:
  excludes:
    - "a"

then buf ls-files will print nothing at all. But if I remove the exclusion, it will complain of duplicates. The inconsistency here is that for the purpose of walking the tree buf considers the symlink and it target to be the same, but for the purpose of linting it treats them as separate different trees.

[bufdev]

There's two things mentioned here:

(1) with build.excludes = ["a"], buf ls-files returns no values. (2) with build.excludes empty, buf build complains of duplicates.

(1) is somewhat by design, to deal with symlink loops.

The relevant code is here: https://github.com/bufbuild/buf/blob/5d03706419e91297ab2738e03de97f9abf359634/private/pkg/filepathextended/filepathextended.go#L62

buf needs to deal with symlink loops when walking a tree structure (similar to any other command that follows symlinks). In buf's case, it keeps track of paths that it resolves when walking a tree structure, and if it hits the same resolved path again, it will ignore it. In this case, it hits a/b/test.proto, resolved to b/test.proto, adds this to the resolved path map, and then when it hits b/test.proto in the walk, it ignores it. Without any build.excludes value set, this results in:

$ buf ls-files
a/b/test.proto

The nature of how buf is built under the hood is that it applies exclusions at a level above - modules are at a higher level of abstraction than the representation of the underlying file structure. So, when you specify a build.excludes = ["a"], this is applied on top of the file structure that has already been resolved. In this case, buf believes that there is a single file, a/b/test.proto, and you are then excluding the entire directory a. So, there are no files left.

We could refactor buf to take build.excludes into account while doing filepath resolution, but we can't promise we could get to this anytime soon. The general principle is that we want to do the best we can with symlink loops within a tree structure, and had to choose the least bad option. But we can think about doing that at some point.

Another command of interest:

# Ignores a, so therefore it sees only b/test.proto
$ buf ls-files --disable-symlinks
b/test.proto

As to point (2), I am unable to reproduce this:

$ cat a/b/test.proto
syntax = "proto3";

package b;

message Foo {}

$ buf build -o -#format=json --exclude-imports --exclude-source-info
{"file":[{"name":"a/b/test.proto","package":"b","messageType":[{"name":"Foo"}],"syntax":"proto3","bufExtension":{"isImport":false,"isSyntaxUnspecified":false}}]}

$ buf lint
a/b/test.proto:3:1:Files with package "b" must be within a directory "b" relative to root but were in directory "a/b".
a/b/test.proto:3:1:Package name "b" should be suffixed with a correctly formed version, such as "b.v1".

$ buf lint --disable-symlinks
b/test.proto:3:1:Package name "b" should be suffixed with a correctly formed version, such as "b.v1".

The overall issue is that there are symlink loops here, and that's difficult for buf to resolve. There is work we could potentially do down the road on (1) however.

[afeinman-snap]

Thank you for these insights. It turns out --disable-symlinks option solves my specific issue. As for reproducing it on your end - I am attaching a repro based on buf linting sample issue_1865_repro.tar.gz Run as:

buf lint acme --config=buf.yaml

[afeinman-snap]

Also, I see from the previous conversation that perhaps I didn't communicate the problem clearly enough. The issue here is that without "--disable-symlinks" linter excludes both link and target if link is specified in build:excludes, but if it's not specified, linter complains about duplicate definitions, which means that when linting, it treats link and target as separate unrelated directories. My expectation is that if it's smart enough to detect that a and b -> a are the same directory, it should also know that a/test.proto and b/test.proto are the same file

[bufdev]

I think doing this detection is outside our scope of allocatable work, unfortunately. Closing this as stale, but can discuss more.