[Feature] Virtual Filesystem

[techman83]

Skyrim's ModOrganizer loads a shim mod into the game which presents all the mods via it's internal virtual filesystem. I'm not sure of the exact details, but there was a discussion about PhysFS today that sounded interesting.

https://github.com/frabert/SharpPhysFS

It would make profiles, mod priority possible and file "overwrites" absolutely possible.

[politas]

So there would need to be a mod running against KSP that intercepted the filesystem calls / implemented PhysFS for KSP?

[techman83]

Essentially yes. It would have to intercept the calls for files and hand them off to the ckan.dll to serve them up. It'd get rid of a lot of the file system shenanigans we have to deal with and we could finally allow overwriting of files.

I have no implementation details, but the idea sounded too cool not to share.

[gajbooks]

I know this is a bit old, but I thought I should clarify the functionality because I believe that this is definitely a feature worth implementing.

I don't believe that the mod manager actually intercepts file calls (unless we are referring to different mod managers). All it does is maintain a list of the mods which have created or replaced a specific file, as well as preserving an order.

When new mods are installed, you can select to overwrite the previous version of a file, and when they are uninstalled, it automatically replaces it with the file which was there previously (from the archive of that mod). If the mod the previous file belonged to was uninstalled in the meantime, it would replace it with the one before that, or eventually, outright delete it when no previous replacements exist.

It would require tracking for every file which is installed with regards to file signature and originating mod, as well as the order of replacement. It would be rude to automatically delete user-modified files, so some check with the mod archive/a hash would be prudent. Some files (configs) could be exempted from this safety check. However, a "clean" feature could refresh the whole directory to pristine mod installs using the saved file orderings.

[techman83]

http://wiki.step-project.com/Guide:Mod_Organizer_Advanced

Mod Organizer takes a very different and revolutionary approach. It never writes files to the game data directory, but instead creates and uses a virtual filing system (VFS). It is called virtual because it does NOT exist on the regular file system and only exists when MO tells a program it exists. When the game is started by MO, it is unaware of this VFS and thinks it uses the regular Windows file system. Therefore, all installed mods are loaded and used normally. The same is true for any other program started by MO (e.g., BOSS, Boss Userlist Manager (BUM), WB, etc.).

That's a really neat approach IMO. We have a lot of troubles due file conflicts and overwrites. This would solve a lot of those.

It was mostly an idea I heard at a conference and if someone had an inkling to write it, this could give a starting point.

[tivec]

I don't believe that the mod manager actually intercepts file calls (unless we are referring to different mod managers). All it does is maintain a list of the mods which have created or replaced a specific file, as well as preserving an order.

Mod Organizer does not handle any form of application read, they build the file system when you hit play from within Mod Organizer.

This would solve a LOT of problems, and would even allow for people to create their own configuration mods where they selectively overwrite files - though, I suppose ModuleManager is already handling this :P

[gajbooks]

I must have been referring to the Nexus Mod Manager rather than Mod Organizer. As interesting as it is to have a virtual file system which doesn't really exist, that feels very fragile for KSP updates and user-written plugins. Despite the obvious speed and space advantage for switching mod sets, that seems a lot more difficult than simply managing file structure.

Once place it would be infinitely useful is for plugins that install their own outdated versions of a plugin over the installed version. If the prerequisites system was used to it's full extent it would be less of an issue, but since the current system involves downloading mods from SpaceDock, many mods still include the DLLs required without any mention of incompatibility.

I would like to think that I could help improve a piece of software I enjoy, but as of right now I have no skills with C#.

[techman83]

I must have been referring to the Nexus Mod Manager rather than Mod Organizer.

Ay :slightly_smiling_face:

As interesting as it is to have a virtual file system which doesn't really exist, that feels very fragile for KSP updates and user-written plugins. Despite the obvious speed and space advantage for switching mod sets, that seems a lot more difficult than simply managing file structure.

In theory since it's only a shim layer for File System calls it should be pretty stable. We already keep a list of files for each mod as far as I recall. And from an installation perspective it would save a lot of shifting of files around (especially if mod profiles are a thing), the zip lib may even be efficient enough to pull the files on the fly without extracting them.

I would like to think that I could help improve a piece of software I enjoy, but as of right now I have no skills with C#.

Even with C# knowledge, this probably requires a good understanding of the code base. But I'm sure there are other issues/features/bugs you could cut your teeth on if your wanting to acquire them. Contributing is a great way to learn! (I didn't know any C# when I wrote some of our NetKAN stuff).

[Dazpoet]

I still don't know any C# and have done a couple of hundred contributions to this project, albeit not to the codebase but rather to the metadata.

[techman83]

Yep, there are a myriad of ways to contribute and the non code contributions are as important as the code :slightly_smiling_face: