timholy
Turn Revise off for a specific folder
Our usecase is that we have a tool that generates Julia files (the Dyad compiler) into a YourPackage.jl/generated folder, whose contents get included into src/YourPackage.jl. The Dyad compiler changes a bunch of files and while it’s working, we would like for Revise to not reload things.
If you happen to run Revise by typing something into the REPL while Dyad compilation is running, you can run into an issue where files that have an include statement no longer exist, or some functions may not exist, because the Dyad compiler has removed them while preparing to overwrite them.
It would be nice to have Revise pick up a .no-revise file in the folder. If it sees this it should not revise anything in that directory. Would appreciate any comments or thoughts you have before we try to implement though.
Some concerns that @StefanKarpinski brought up were:
what happens if there are multiple processes that are trying to do this
Any process that writes Julia code could be thought of as a compiler of a sort. You wouldn't run cc in multiple windows, simultaneously, on the same directory, and expect a sensible result. So this can be considered undefined and bad behaviour.
More generally, any software should detect the presence of a .no-revise and be smart enough to handle that if it sees it.
what happens if deleting the file doesn’t happen? Then you have this file laying about and it never gets deleted and revise effectively stops working until someone/something deletes it.
The process that created the .no-revise would have to handle it and delete it. Barring segfaults etc. this seems broadly doable.
If this doesn't make sense or there are other solutions, would love to hear about them! For what it's worth this is running in a separate process, so I don't think we can use #943 here (though that would have been a great solution if we could).
or there are other solutions, would love to hear about them!
Would a similar effect be achieved if you debounced the FileWatching events? A debounce of 1-2 seconds would probably be good enough for your usecase, especially if you create your files somewhere and "atomically" copy them over (atomic copy = rename in the same device)
The delays in file generation can be longer than that though. And there is never really an atomic operation on the filesystem...especially if you have 1k+ files.
And there is never really an atomic operation on the filesystem...especially if you have 1k+ files.
Moving an inode is atomic, with these guarantees https://man7.org/linux/man-pages/man2/rename.2.html.
So, dyad compile -o /tmp/new; mv pwd/generated /tmp/trashcan; mv /tmp/new/generated pwd/generated is a series of two atomic operations (given the filesystem doesn't crash and that /tmp is in the same physical disk as pwd). Should be nanoseconds.
I'd suggest flocking the lockfile instead of just checking for its presence. That neatly addresses the concerns, in that
- multiple instances of the dyad compiler can actually check whether they are working on the same directory
- the lock gets released if the locking process stops
Any process writing to the directory (the dyad compiler in this case) would take an exclusive lock on the lockfile, whereas any julia process running Revise will take a shared lock on it.
And there is never really an atomic operation on the filesystem...especially if you have 1k+ files.
Moving an inode is atomic, with these guarantees https://man7.org/linux/man-pages/man2/rename.2.html. So,
dyad compile -o /tmp/new; mv pwd/generated /tmp/trashcan; mv /tmp/new/generated pwd/generatedis a series of two atomic operations (given the filesystem doesn't crash and that/tmpis in the same physical disk aspwd). Should be nanoseconds.
That does seem like a strictly superior implementation, but it's probably not 100% reliable in the presence of NFS...
Would https://timholy.github.io/Revise.jl/stable/user_reference/#Revise.dont_watch_pkgs work?
But if you only want to transiently prevent revisions, then I think @pfitzseb's suggestion seems much better.
Flock seems ideal... but doesn't that also not work on NFS? My impression was that PidFiles was designed to handle this kind of thing reliably.
@vtjnash can you comment on whether PidFiles is the right tool here?
Yes, PidFiles is designed specifically for this. Flock is a kernel implementation of something similar, but much more limited and not as good, since it is stuck inflexibly in kernel space instead of in user space.
atomic rename is also far better than any sort of lock, but not always possible for various reasons
Ok, I think what we should do here is add support for .no_revise files using PidFiles—if the file exists, don't Revise anything in that directory or below. Dyad can also try to make file updates more atomic but may not be feasible.