fastly
infinite rebuild loop using compute serve --watch
Version
fastly version (base) (story/CORE-734*) 14:01:01 fastly fastly version (base) (story/CORE-734*) 14:01:01 Fastly CLI version v10.15.0 (0a1665b5) Built with go version go1.22.7 linux/amd64 (2024-10-30) Viceroy version: viceroy 0.12.1
What happened
Happens:
While running fastly compute serve --watch we ended up with an infinite loop of rebuilding, it seemed that something on the specific Mac, that was used, was doing a chmod operation as soon as fastly compute serve was issued creating an endless loop.
I debugged the fastly cli code and saw that it was always the fsnotify operation CHMOD.
Digging deeper into the code and into fsnotify it's stated in their documentation that taking action on Chmod isn't recommended.
See their code base: https://github.com/fsnotify/fsnotify/blob/2d133b81be1e26628491698a699ef9ce9fb71ed9/fsnotify.go#L135
Our theory is that company "must have" software (anti-virus) is getting triggered by the fastly compute serve operation
and that starts this loop.
Expecting: Rebuild shouldn't occur if I haven't actually changed the files (write, create, remove, rename)
Temporary solution:
I patched the code by adding a if statement to only allow the event handler if the event had either Create, Remove, Rename, Write, which would exclude if it was only a Chmod operation.
I'm not sure if that's a good solution in the long run, maybe an cli option to control the behaviour of watch would be better
Thanks for the excellent report! I agree with your findings and conclusion; the permission bits (mode) of the file are not relevant for triggering a rebuild, and it makes sense to only pay attention to the other modifications. Based on the comment in that link there are some non-mode changes which can also send a CHMOD event, but I don't see them as being a big concern (one is removal of a hard-link, and the other is truncation of a file).
If you want to submit a PR with your suggested changes to this logic, it would be very welcome.
I have always had issues with fastly compute serve --watch running builds 2-3 times before finally settling into a state where I can use the server. I used to just figure it was a quirk of the CLI and patiently wait for it to settle down. However I just picked up a Compute project after not touching it for a few months, and the problem seems to be many times worse. I upgraded Viceroy from 0.12.3 to 0.14.4 and Fastly CLI from 11.5.0 to 12.1.0, and now my project builds 8-10 times before settling. It ran so often that cargo clean cleaned up 2.2 GiB of files when I was finished!
The CLI seems to restart the local server at random. It references files that have not changed, and sometimes references files that should be ignored like those in target/. I had a local server running while writing this comment, and already it has restarted a dozen times even though I haven't touched any files. Here's some samples from the output:
✓ Restarting local server (/project-directory/target/wasm32-wasip1/release/libspeedcurve_rum_beacons.d)
✓ Restarting local server (/project-directory/target/wasm32-wasip1/release/deps/potential_utf-efe3276bcbe8d0ea.d)
✓ Restarting local server (/project-directory/fastly.toml)
Edit: I will note that --watch-dir does seem to work, but it is annoying when I change things in (for example) the root directory.
As it turns out this isn't directly a problem in the CLI itself, but the lack of a .fastlyignore file which tells the CLI which directories/files it should not watch for changes. Until the summer of 2023 the CLI automatically used .gitignore for this purpose, but that was changed and no replacement was put in place... and the result is the CLI watches directories which contain temporary and output files when it should not. @harmony7 is going to be adding .fastlyignore files to our starter kits so that at least newly-created Compute projects won't suffer from this problem, but we'll also need to send out guidance to our customers who have existing projects to let them know what they should add in their own source directories.
If you want to experiment with this yourself, you can add a .fastlyignore file containing /target (for a Rust project) which will stop the CLI from looking inside the target directory.
Ahh right I did see #555 and thought the .gitignore functionality might be broken. I will try out .fastlyignore - thanks @kpfleming!