AcademySoftwareFoundation
Hooks for package caching?
I would like to, somehow, make the pkg-cache directory read-only - so packages can be added and removed, but their contents not altered by the user
Motivation For example a Rez package might contain a template file someone might open for reference, but they should not be able to accidentally save over it.
We have a post-release hook which removes write-permissions from released packages. This helps prevent accidental modifications to installed packages.
However if the package package is cached, this gets copied to a writable location, so it's possible to accidentally change something, which can be equally confusing
What OS are you on? Rez uses shutil.copytree(), which is supposed to copy permissions (via copystat: https://docs.python.org/3/library/shutil.html#shutil.copystat)
Oh, good point - I forgot to mention this is on Windows, using ACL, so things aren't so simple :hurtrealbad:
Specifically the release hook runs something like icacls.exe $folder /deny DOMAIN\group:(OI)(CI)(W,D,DC,WD) on each variant at release time
shutils.copytree() is very basic on how it copies files and just handles Posix permissions and symbolic links. It does not handle things like extended attributes (only supported on Linux), ACLs, and dual fork files. In an MacOS environment extended attributes become important when you start dealing with code signing on shared storage. In a Windows environment ACLs are needed for basic file permissions.
Since packages are copied as the user currently running the rez command you can end up with interesting permissions problems on a shared machine (multiple shifts, renderfarm, ...)
If some sort of hooks are added to package copying we would also most likely need hooks for when a package is cleaned from the local cache.
Just to put this out there without having really thought it through. I was wondering in the past if the actual copying should be pluggable like VCSs. That would also open up rather easy ways to implement things like zipped payloads nor non-filesystem sources maybe
We've also been interested in locking down package caches, which led me to this issue.
But since I'm here...
Just to put this out there without having really thought it through. I was wondering in the past if the actual copying should be pluggable like VCSs.
@instinct-vfx we have actually implemented this feature (a copy_process plugin type) internally. I believe there has always been the idea of contributing it back, so I'll dust that conversation off and see if that's still on the table.
@nrusch +1 I would be most appreciative to see your implementation for a copy_process plugin
I'm currently investigating slow copy speeds on pkg caching, I've previously use Rez on linux and now in a windows env and windows is slow.
I've been looking at how shutil.copytree works under the hood, and determined that Linux and Mac use C lib binds to optimise copy operations while in windows its done in python. https://github.com/python/cpython/blob/3.13/Lib/shutil.py
There was a python issue about including binds to windows's TransferFile syscall similar to the linux os.sendfile but no progress has been made on that it seems. https://bugs.python.org/issue21721 due to maintains not sure how best to translate winApi file handle.
So the only method I know available was to test against pywin32 win32api.CopyFile which produces a sizeable speed increase, one test example below. Added to copytree, ie shutil.copytree(src, dest, copy_function=CopyFile)
The test is done on Window10 with Python3.10, copying from a SMB share to local disk C:, the Linux test is done on the same host but from a WSL Alma9, mounting the same share with cifs-utils, copying to the WSL filesystem which is the same drive as window local C:.
This is to minimize disk/network differences, The same pkg is copied each time.
I did the test on different payload sizes, ill share with you the small size numbers for 50~MB, 6442 files.
Windows copytree time: TIME: 92.3725426197052
Linux copytree time: TIME: 32.71613907814026
Windows copytree time with pywin32 CopyFile as copy_function: TIME: 15.157870054244995
Shell invoked Robocopy: Total 23s, Copy 13s
I think ability to customise filesystem copy would be very welcome (i know i will), it might also play into the remote repository plans
@vanridal It sounds like there are no internal blockers to submitting the plugin implementation upstream, so I'll be looking to get a PR up as soon as I can.