Virtual contexts: Multiple file systems in an isolated mount tree

[jeff-hykin]

Thanks for all the work on ZenFS! I'm hoping to use it as a platform for a lot of projects.

To partially continue the conversation from #124 , my big question is: what variables/functions/classes are stopping ZenFS from being able to load two different backends and maintain them as separate file system trees.

I understand the challenges with Nodejs, and not having a giant ZenFS class. If I attempt a fork ZenFS, I'd attempt to use a binding technique rather than a giant class. And, well, it seems like V_Context is doing a lot of what I would've attempted. So I'm working on understanding what is missing from V_Context that limits it to just one file system tree.

To be clear on expected behavior; I expect backends like NodeFS to only ever have one file tree. In-memory trees that exist along side a global file system is the question for this issue.

Trying to answer that myself:

Other than the mounts variable in src/vfs/shared.ts, I don't see much that would prevent ZenFS having two file system trees.

That said, I imagine it is more than just mounts. After drowning in the complexity of the config.ts mounting/unmounting, I figured I should just ask how big the iceberg is.

[james-pre]

Hi Jeff, thanks for your interest in the project.

The single tree of mounts came from architectural requirements. Specifically, for Node.js compatibility, ZenFS has named exports for most FS functions (e.g. import { statSync } from '@zenfs/core'). This necessitates using functions (rather than methods) and prevented any kind of internal state shared between functions.

Originally, different contexts were not supported— the functions all shared some semi-global state (including PWD, descriptors, and credentials). This also lead to much of the VFS using a functional approach, since this is cleaner and easy to maintain (especially considering the lack of internal state).

Contexts are, in essence, a creative way to share internal state among separate functions by binding an object to the function's this value. This means import { exampleSync } ... works correctly, event when the imported function has some state associated with it. If you read through the VFS code, you'll notice a significant amount of .call(this, ...)— this is how functions are able to pass the state around without breaking compatibility with the node:fs API.

But how does this relate to multiple mount trees?

A little function called resolveMount was added before contexts. It goes through the current mounts and finds the one responsible for a given path. When contexts were added, resolveMount was extended to work with context roots. This function uses a Map called mounts to track what file system is mounted where. The mounts map is shared among contexts, meaning they all share the same set of mounts (though the root can change, meaning some mounts may not be visible to some contexts).

Now that you've brought this up, it would be theoretically possible to store a mounts map on contexts. Then, one could create a new context tree with its own mounts. This would introduce more complexity into configuration though, and could have a performance impact.

However, it is important to consider that this might not be needed— contexts already provide a way to isolate parts of the FS tree. They've been designed as a way for the "host" code to create a security boundary within the existing tree, preventing code given the bound context from escaping.

For example, you could leverage this with Compartments by exposing only the bound fs of a context to a compartment. (note the ses package polyfills this)

Please let me know if you'd like more information or if you have any questions.

[jeff-hykin]

Thanks @james-pre for such a detailed response! Your note about compartments is really interesting, I'll be looking into that for some of my other projects.

I think multiple trees still has valid usecases (ex: two libraries internally using ZenFS, without the application dev even knowing they use ZenFS).

So I'll try making a fork and putting the mounts inside the context. That should help me get an idea of the complexity and possible problems, then I can report back. (Might take a few weeks)

I'll probably:

1. add an argument to bindContext() that let's it use a separate mounts map.
2. Have bindContext output a (bound) configure function that allows it to get connected to a different backend

[james-pre]

I think multiple trees still has valid usecases (ex: two libraries internally using ZenFS, without the application dev even knowing they use ZenFS).

Most programs work fine using a single tree. Perhaps it'd be better for these libraries to follow the established way of doing things (e.g. using /etc, /bin, and /lib). This allows libraries to be deduplicated and stuff. After all, one of the goals of this project is to emulate Linux's file system behavior.

I'm still unsure of whether this is a feature that would be worth integrating, especially given the minor perf impact and more importantly the maintainability impact.

Anyway, here are some tips for your implementation:

I'll probably: add an argument to bindContext() that let's it use a separate mounts map.

I'd recommend you extend ContextInit, since that allows for future changes without breaking the API.

Have bindContext output a (bound) configure function that allows it to get connected to a different backend

This could introduce a ton of complexity. I'd recommend just adding a mount and umount function directly to the bound context (but not it's fs property). This is a lot less complicated, and for such advanced use cases, configure may not be as useful anyway.

Hopefully this has been helpful!

[jeff-hykin]

This could introduce a ton of complexity.

Good news!

• less than 20 lines of src/ changed
• no changes to the existing arguments (only additional optional attributes or args)
• all tests (including new isolation tests) are passing!
• no measurable perf impact on tests

Conceptually it pretty much was as simple as ctx?.mounts || mounts, which was surprising to me. Also yeah, after understanding the code base better I agree returning a bound configure function would have been a terrible design.

I'll refine/check it more before making a PR, but I'm starting to think it won't conflict with your goals after seeing how nicely it fit into ctx and configure.

I'd also like to offer what help I can too. I noticed some of the tests are failing on the latest main branch, some awaits were missing in front of some assert.reject's and rmdir non-empty test seems flakey (at least on the 2.0 release commit). If you think it'd be useful I can try to get the tests passing and maybe expand the existing test cases.

Perhaps it'd be better for these libraries to follow the established way of doing things (e.g. using /etc, /bin, and /lib)

Oh, I think there's a big gap in our perspectives! None of the libraries I had in mind use any of those directories. Hopefully I can communicate the view I have, as I think its not nearly as advanced/niche as a use case like chroot or permissions or even just mixed backends on a single tree.

Most programs work fine using a single tree

If you said "most packages work fine with a single tree" I would agree immediately. But one program can have hundreds or thousands of packages.

What if the host doesn't even realize ZenFS is in their dependency tree? I'd argue most npm projects have more indirect dependencies than direct dependencies.

Imagine a tree like:

• ZenFS (leaf)
• package Z (using ZenFS)
• my software (using Z, could be nodejs or in-memory)
• package A using my software (only uses in-memory case)
• package B using package A (hides the FS setup of pkg A, there's no longer a way to configure or pass-in a fs)
• package C using package B (ex: a web component that renders the tree)
• host user (using only package C)

Or maybe a better example would be this:

1. The host doesn't want to deal with file trees, they just want to give the component a url and have number-of lines number of commits, etc visualized
2. The host couldn't give a central file tree to both tools (webgit and websvn) because the react components don't expose an interface for doing it
3. WebGit and SebSVN don't know about each other, it doesn't make sense for them to try and share a file system

Making standalone tools like unzip or isomorphic git, or image converters, or esbuild-wasm doing client side bundling on CodePen, etc. they just need a single isolated folder, not a /lib and /etc or a complex sub-tree system.

And in terms of manual isolation, the workarounds like duplicate ZenFS or Compartments have major downsides. Like platforms that don't have a file system, like CodePen or Val.town, where an isolated ZenFS would be great, don't work with Compartments since ses either needs a build step or has major runtime cost.

[james-pre]

Jeff,

I just wanted to say thanks again for your interest in the project, It's great to see so much community involvement.

Just FYI the current state of main is not golden (though it should be) due to #210. It's been a huge amount of work for such a seemingly minor issue. I'm still working on fixing some bugs that have arisen from the work done on that issue. These are the bugs currently causing tests to fail. My current priority is getting main back to a stable state, then I can take a look at merging any PRs.

You argue some compelling use cases. My reasoning against having separate mount trees has been based on comparing web programs to native ones. The web is very different, and I understand that use cases may want a different environment.

If you want to open a PR, that's fine. My current thoughts is that contexts would have a new mounts: Map<...> property, which is automatically inherited. ContextInit can have an optional mounts property that overrides the inherited one (and explicitly does not merge with it), giving the newly bound context a completely independent mount tree. You will also likely want to modify the existing mount and umount functions to have this: V_Context if they don't already.

By the way, I noticed your comment on assert.rejects and quickly made the change myself— I'm surprised I missed this when writing tests, nice catch! I'll hopefully be done fixing main soon; I just finished fixing a couple of very sneaky upstream bugs that I literally spent over 50 hours trying to find.

[jeff-hykin]

All good!

I'll just wait till main is stable again. I'm not in a rush.

I've fought with SharedArrayBuffer a couple times so I feel your pain with #210.

By the way, I noticed your comment on assert.rejects and quickly made the change myself

Great!

Speaking of changes that are easier without a PR. Could you try changing #!/usr/bin/bash to #!/usr/bin/env bash (at the top of tests/fetch/run.sh)? If that replacemnt works on your end, it would help me out cause the current hashbang doesn't work for me on MacOS Big Sur.

My current thoughts is that contexts would have a new mounts: Map<...> property, which is automatically inherited. ContextInit can have an optional mounts property

Haha, that exactly what's on my fork! Seems like we're on the same page. There are a couple details like addDevice, but I'll save that discussion for the PR.

[james-pre]

Hey Jeff, main is now stable and v2.1.0 is out. Also, I updated the hashbang. Please let me know if you have any questions.

[kylecarbs]

@james-pre could you provide an example of how to do this? I've tried to follow this thread, but an example would be great.

[james-pre]

Hey @kylecarbs, apologies for the delayed response.

I think the addition to the API would look like this:

const ctx = bindContext({
    mounts: new Map([
        ['/', InMemory.create({})],
    ]),
});

ctx.fs.mkdirSync('/ex');
const example = await resolveMountConfig(...);
ctx.mount('/ex', example);

Please note that this would be for advanced use cases that aren't satisfied with existing APIs (including contexts).

[jeff-hykin]

Hey Jeff, main is now stable and v2.1.0 is out. Also, I updated the hashbang. Please let me know if you have any questions.

Dang I missed this notification! I'm glad there were a few more comments here. Hopefully next week I'll have a good PR for this.

[james-pre]

Hey @jeff-hykin!

I was thinking about this issue some more, I think this API would keep things clean and separate:

• Keep mounts only on FSContexts
• The existing "global" mounts can be moved to the defaultContext
• Each set of mounts has a different context tree, and also different context IDs
  • This creates a clear security boundary, so programs can't gain information about what other programs might exist.
• bindContext and BoundContext.bind only work within an existing context tree
• A new function, maybe something like createContextTree is the way to create a new tree of contexts (with different mounts)

I came up with this in about 15 minutes, criticism is welcome. Please let me know what you think about this, especially if you think it is over-engineered.

[gwsbhqt]

Hey @kylecarbs, apologies for the delayed response.

I think the addition to the API would look like this:

const ctx = bindContext({ mounts: new Map([ ['/', InMemory.create({})], ]), });

ctx.fs.mkdirSync('/ex'); const example = await resolveMountConfig(...); ctx.mount('/ex', example); Please note that this would be for advanced use cases that aren't satisfied with existing APIs (including contexts).

We need this feature, keep going!

[jeff-hykin]

@james-pre I was working on this today, made a lot of progress, but when I went to test there was a problem. Even on the main branch I get a lot of failing tests. Is this normal?

[james-pre]

Hey Jeff,

All the tests on main are currently passing. You can try pulling and clean installing dependencies. Also make sure you have at least Node 22 (the current LTS) and use regular npm. Other than that I can't think of anything tooling or dependency related that would cause issues. You probably just have some bugs in the changes.

[jeff-hykin]

Hey Jeff,

All the tests on main are currently passing. You can try pulling and clean installing dependencies. Also make sure you have at least Node 22 (the current LTS) and use regular npm. Other than that I can't think of anything tooling or dependency related that would cause issues. You probably just have some bugs in the changes.

Thanks! and no problem, I'll figure out whats wrong with my system and then get back to it. Just wanted to confirm it was me.

In the meantime here's an update!

• Keep mounts only on FSContexts
• The existing "global" mounts can be moved to the defaultContext

This worked great. After I implemented it, I can confirm its a lot cleaner.

• Each set of mounts has a different context tree, and also different context IDs

This also worked nice.

• I created a separate array of all roots (e.g. one element per context tree) and I changed the context ID's into strings that make a hierarchy.
  • tree 1 (AKA the default context) still has id 0
    • child 1 of tree 1 has id of 0.0
      • grand child 1 of tree 1 has id of 0.0.0
        • etc
  • tree 2 has an id of 1
    • child 1 of tree 2 has id of 1.0
    • etc

• bindContext and BoundContext.bind only work within an existing context tree

• A new function, maybe something like createContextTree is the way to create a new tree of contexts (with different mounts)

Also worked great. Took me all day, but I converged on the following:

import { context as defaultContext, createContextTree } from '@zenfs/core';

// child of existing tree
const { fs: childFs } : BoundContext = defaultContext.createChildContext({ root: '/1' })

// separate tree
const { fs: isolatedFs } : BoundContext = createContextTree({ root: '/' })

Breaking up bindContext into two cases (child context and new-tree) made the logic much easier to handle conceptually. On the interface side, I think hiding bindContext completely and only exposing the method (ex: context.createChildContext) is the way to go. I export/expose a bound version of the default context.

I also did an important change to V_Context. When random parts of the code access the default context object, it is a bit prone to context-handling errors. For example, I think normalizePath has a bug where the context isn't taken into account correctly (it always used the default context instead of the active V_Context). It was hard to know/think about every case of default-ing logic. So to simplify it V_Context is now either null or a FSContext (no longer a partial object). To make sure that was strictly enforced there's now a small helper getContext. Logic like this?.root || defaultContext.root has been standardized to getContext(this).root (or getContext($).root). This made it a lot easier to think about, because, after that change, no files needed the defaultContext directly. E.g. now that the global mounts has been put inside defaultContext, its basically confirms there's no lingering context issues, everything must to access methods via getContext($), cause there's no global mounts var or root etc. Well ... almost ... _setAccessChecks is a problem I didn't get to, but I think that's the last global-ish var.

[james-pre]

Hey @jeff-hykin, hope you're doing well.

In light of #246, I'd like to discuss the future of this very cool feature with you.

Before I get into that, I want to say this will be a solid feature to add to the existing v2 branch once v3 is done.

My first thought is that this is very similar to namespaces, and would likely carry over similaly. For v3, would you be interested in working on namespaces?

For an overview on namespaces, these pages are useful:

• Linux namespaces - Wikipedia
• namespaces(7)
• pid_namespaces(7)

Feel free to drop a comment here, send an email, or reach out on Discord— the latter would provide faster and less formal communication.

[jeff-hykin]

Thanks for the invite! I'll probably jump in the discord at some point. I actually do like long github messages as a way of refining my thoughts though.

I'll keep working on it for v2. Good news is, I absolutely share your vision for v3. I want a node-fs interface for, let's say, a linux emulator using only in-ram memory. And then I want the same node-fs interface for, let's say, git-wasm, which uses IndexDB storage to let people explore git repos in their browser.

That said, I think there is a divergence on this issue, one that I think could really help with the design of v3.

There's a complexity gap between this issue and Linux namespaces / bound child contexts:

• Multiple isolated file trees is conceptually super simple: an fs object is bound to one (and only one) context tree, it is always bound to the same tree, it shares nothing. Doesn't require multi-backends, mounts, child contexts, permissions, anything. Its just the idea of isolation. Which is why I keep saying "this feature is not an advanced usecase". I cannot start using ZenFS until it supports simple isolation.
• By contrast, I agree, bound child contexts and/or Linux namespaces is an advanced feature. Its like a rubix cube; changes on one tree can effect another tree. I do eventually want to help add that feature, its important. I definitely want ZenFS to be forward-compatible with that capability. But it feels a bit wrong to work on the advanced case when the simple case isn't supported.

The good news is, I have an idea for v3 that might address both: VirtualStorage. (You might be ahead of me on this)

• Right now AFAIK each backend exports itself as a file system (ex: Store extends FileSystem). As of now, I think that design is a subtle mistake that makes the mental model difficult. Linux has multiple "backends" (hardware) but its not like a hardware component can have a mount. The mount is a kernel concept, its not owned in the backend (hardware) itself.
• So imagine something inbetween a backend and file systems: Iets say a VirtualStorage interface.
  • There's two sides of VitrualStorage: the kernel perspective, and the ZenFS perspective
  • 1. From the ZenFS perspective, VitrualStorage would be like a "base case" of file systems. VirtualStorage is the standard interface of backends. A virtual storage object must be one-to-one with a backend. That means it couldn't have permissions (kernel concept), wouldn't have mounts, and any error from a VirtualStorage would be treated like a hardware error, not an OS error. The virtual storage object might have an API that looks like a file system, but internally we treat the object like its raw storage.
  • 2. From the Kernel perspective, a VirtualStorage object would represent hardware, like a flash drive or an SSD. The real/full file system created by the kernel would be built on top of different kinds of VitrualStorage devices. The kernel would manage concepts like permissions, Linux namespaces, mounts, etc.

I think that split alone could address this thread (context isolation), because, for many simple applications, I would just use VirtualStorage directly. Most people don't need multi-backend mounts, permissions, child contexts and so on. I just need to be able to create two in-memory file trees that aren't connected.

At the same time, this change could be a good step towards supporting a kernel-managed filesystem with better device files, Linux namespaces, etc.

The names VirtualStorage and FileSystem could probably be renamed to SimpleFs and AdvancedFs.