pantsbuild
Retry mechanism restarts spuriously
The log message says it is waiting 500ms, but based on the logs it doesn't seem the case...
Or it might just be the case that we log this message for every target.
In this case, I am running ./pants binary on around ~50 targets.
At any case, this is bad UX that should be improved.
The way to trigger this is to run a command that rebases a branch (git rebase upstream/master for example).
@stuhood @gshuflin @Eric-Arellano Follow up on https://github.com/pantsbuild/pants/issues/10518 - there is progress... but still not good enough IMHO.
also noticed that when running the fmt goal it triggers this message (when fmt actually changes files):
20:10:54.36 [INFO] Filesystem changed during run: retrying Fmt in 500ms...
Relates to #9462: in order to support django runserver and live reloading repls we should decide which cases do want to restart for file changes, and which don't.
Related to this issue (but probably not worth it's own ticket quite yet) is the fact that this message is printed when a node is dirtied, even if it might be cleaned later (see Dirtying and Cleaning)... which means that many of these messages are actually spurious: there might be no need for us to interrupt the @rule if it turns out that its dependency will be cleaned.
Related to https://github.com/pantsbuild/pants/issues/10705#issuecomment-722757217 ... when these restarts are spurious, they're extra frustrating (had another report today), because they can be triggered by completely unrelated files.
Thought a little bit about how to fix this, and the first thing that came to mind was adjusting the clearing of nodes (in invalidate_from_roots) to trigger "clear and eagerly re-run" rather than just clearing, and to move the dirtying of dependencies to after we've determined that a node has actually changed generations. It's possible that invalidate_from_roots could become async to drive that process itself.
An assumption in that implementation is that the nodes that we invalidate (Nodes with associated filenames) are always tiny (individual syscalls or the snapshotting of single files) rather than processes running, etc.
We're still seeing this with Pants 2.12, and it seems like a potential route for improvement that's not finer-grained invalidation would be to ignore changes in SCM directories like .git by default, because that's definitely a source we're seeing of these:
11:58:30.84 [INFO] handling request: `--pants-bin-name=./pants --pants-version=2.12.0 test ::`
...
11:58:39.54 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".git", ".git/index.lock"}
11:58:39.54 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".git/index.lock", ".git"}
11:58:45.32 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".git", ".git/FETCH_HEAD"}
11:58:45.32 [INFO] notify invalidation: cleared 1 and dirtied 2096 nodes for: {".git", ".git/FETCH_HEAD"}
11:58:48.15 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".git/FETCH_HEAD"}
11:58:48.15 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".git/FETCH_HEAD"}
11:58:48.39 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".git/objects", ".git/objects/maintenance.lock"}
11:58:48.39 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".git/objects/maintenance.lock", ".git/objects"}
11:58:48.39 [INFO] notify invalidation: cleared 1 and dirtied 2096 nodes for: {".git/objects", ".git/objects/maintenance.lock"}
11:58:48.39 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".git/objects", ".git/objects/maintenance.lock"}
...
11:58:59.98 [INFO] request completed: `--pants-bin-name=./pants --pants-version=2.12.0 test ::`
We've been able to work around this via pants_ignore = [..., ".git"].
Hey @huonw, .git is ignored by default. You might be encountering a really common gotcha: did you use pants_ignore = [] rather than pants_ignore.add = []? If so, that will override Pants's default of .* :/
Yea, as @Eric-Arellano said: .git should be in your pants_ignore by default. Use .add when adjusting that.
I've started to see this issue as well:
./pants --print-stacktrace generate-lockfiles --resolve=black
11:42:49.05 [INFO] Initializing scheduler...
11:42:49.39 [INFO] Scheduler initialized.
11:42:54.99 [INFO] Completed: Generate lockfile for black
11:42:55.00 [INFO] Filesystem changed during run: retrying `GenerateLockfilesGoal` in 500ms...
11:42:55.51 [INFO] Filesystem changed during run: retrying `GenerateLockfilesGoal` in 500ms...
11:42:56.04 [INFO] Filesystem changed during run: retrying `GenerateLockfilesGoal` in 500ms...
11:42:56.55 [INFO] Filesystem changed during run: retrying `GenerateLockfilesGoal` in 500ms...
...
In my pants.toml file, I've got the following:
[black]
version="black==22.6.0"
lockfile = "/3rdparty/python/black.lock"
Unsure if I'm doing something wrong on my side.
Currently running pants_version = 2.14.0
[SOLVED]
Turns out I needed to specify a relative file path e.g
[black]
version="black==22.6.0"
lockfile = "./3rdparty/python/black.lock"
Then it worked 🚀
./pants generate-lockfiles --resolve=black
11:53:53.16 [INFO] Completed: Generate lockfile for black
11:53:53.17 [INFO] Wrote lockfile for the resolve `black` to ./3rdparty/python/black.lock
