Proof of concept: RattlerFS

[chrisburr]

This add a proof-of-concet FUSE based filesystem that transparently proxies the data in the package cache on demand.

Inspired by https://github.com/conda/rattler/pull/1162, read there for some of the motivation. Unlike the overlayfs approach, I think this could work on linux, macos and even windows.

Also this is my first time writing rust so it's a bit of a mess but it shows the idea:

$ cargo run --bin rattler_fuse_test -- pixi.lock /tmp/cburr/test --env-name default

I would imagine this would normally be exposed by some kind of config setting for tools like pixi such that pixi run xxx sets up the fuse mount, runs the command and then exists. This works but it does wipe out the kernel filesystem cache every time, so there would be an argument for having a pixi mount command or similar.

The main missing functionality is:

• [ ] Compilation of pyc files
• [ ] Post-link scrips
• [ ] Windows entrypoints
• [ ] pip packages
• [ ] The conda-meta/ directory
• [ ] Nice-to-have: Include the offsets in the JSON files of the package cache to avoid the need to seek through files to find the instances of the prefix.

Post link scripts (and pyc files) could be implemented either with overlayfs or by making RattlerFS writable (presumably as an overlay to another directory).

.pyc files could alternatively be done by adding them to the central package cache and compiling them on demand.

And if anyone would like to take over this I'd be happy to let them 😄

[baszalmstra]

Very exciting!

[matthewfeickert]

@chrisburr :wave: Following up from our discussions in March, just curious if you have any plans for the future of this?

[chrisburr]

In September I'll have an intern who will be working on this.

[baszalmstra]

For Windows I was looking into ProjFS, which provides a similar mechanism as fuse but natively for Windows. We could also do something similar with NFS, maybe. Perhaps it makes sense to set up this module in an abstract fashion where we can implement adapters for different filesystem bridges.

[chrisburr]

Perhaps it makes sense to set up this module in an abstract fashion where we can implement adapters for different filesystem bridges.

Yeah abstracting this definitely makes sense! I think FUSE makes sense for the first implementation as:

• It can easily be mounted without special privileges on Linux (IIUC using NFS would require root to set up)
• There are compatibility layers for macOS and Windows

In the longer term there is:

• ProjFS for windows
• FSKit for macOS
• NFS could be interesting for macOS or as a way of sharing environments over a cluster without having a dedicated mount on each machine

In the next week or so I'm going to open an issue with plan for implementing rattler-fs (or whatever we call this before my intern starts in September.

[baszalmstra]

Makes sense! Id be very happy to contribute in any way I can.