FileSystem proposal

[pierrec]

trafficstars

I propose adding a new FileSystem type in the std::io module to abstract how files can be accessed and have a default implementation for the operating system file system.

One use case is about serving content for various APIs regardless of where the files are coming from (local fs, embedded zip file, etc).

module std::io::fs;

/*
 * FileSystem is a read-only filesystem containing either files or directories of files.
 */
struct FileSystem
{
	FileSystemInterface fns;
	void* data;
}

def RootFileSystemFn = fn String (FileSystem);
def OpenFileSystemFn = fn FSFile! (FileSystem, String);
def OpenDirFileSystemFn = fn Path! (FileSystem, String);

struct FileSystemInterface
{
	RootFileSystemFn root_fn;
	OpenFileSystemFn open_fn;
	OpenDirFileSystemFn open_dir_fn;
}

fn String FileSystem.root(self)
{
	return self.root_fn();
}

fn FSFile! FileSystem.open(self, String name)
{
	if (OpenFileSystemFn func = self.fns.open_fn) return func(self, name);
	return IoError.UNSUPPORTED_OPERATION?;
}

fn Path! FileSystem.open_dir(self, String name)
{
	if (OpenDirFileSystemFn func = self.fns.open_dir_fn) return func(self, name);
	return IoError.UNSUPPORTED_OPERATION?;
}

// An FSFile represents a file in a FileSystem.
struct FSFile
{
	inline Stream file;
	String name;
}

fn bool FSFile.is_dir(self)
{
	return path::is_dir({ self.name, PathEnv.POSIX });
}

fn bool FSFile.is_file(self)
{
	return path::is_file({ self.name, PathEnv.POSIX });
}

[lerno]

The reason why File is a struct rather than a distinct wrapper around a CFile is because I wanted to keep the door open for using it universally, like your "FSFile" here. Being a struct it could fairly freely get extended with additional data etc.

So the reason I didn't do anything further (yet) is because I didn't have any code that needed it (yet).

So my thinking is that if one tries to create a generalization of some functionality and you just have one variant when you start - then this will fail for sure because it's not possible to figure out a good generalization. With two usecases it's possible to get something ok if one is lucky, but what you want is at least three different use cases to generalize from.

So before we have these other variants, designing the generalization of a File will probably be hard to do well.

[pierrec]

OK so a FileSystem would be one use case. Let's see what else would be required.

[pierrec]

BTW I appreciate your thoughtfulness in how you approach your work and careful planning!

[lerno]

I've tried to do it the other way around too many times that I've learned my lesson.

On a related note: one thing that often strikes me is how game frameworks often have completely different style of APIs for getting data from files than the usual standard library APIs. Often they are more focused on getting the job done. Typical things are like "read this entire file and return the bytes", which are surprisingly often missing from file system APIs in their first iterations. So I think looking at how to make it practical to use files and not just focusing on giving the barebones full functionality is an important thing to study.

[OdnetninI]

I have also encountered two other use cases while working with some scientific benchmarks:

• Read/write the nth structure of a file. This is typically done by calculating offsets, fetch and read (only works for static size structures), or even read completely and then map to the correct structure.
• Another case I found was to hierarchize a complex data structure into files and folders. Like a b-tree. I have seen this once for a very large structure that was not able to fit in memory, so this was the only solution.

However, I think use cases will appear when more programmers get involved in the language, or when more libraries/applications from different areas are ported.

[DrGo]

Go has a neat generalization of file systems that is used widely e.g., for mocking/testing, implementing specialized file systems e.g., ones that restrict access to certain dirs, etc. Essentially any tree structure can be represented as a file system allowing use of all builtin filesystem functions e.g., visiting every node.. See https://pkg.go.dev/io/fs

[lerno]

@DrGo I've read people being critical about Go's file handling in general due to it being strongly unix-centric. What are your thoughts on that in relation to fs?

[DrGo]

File operations in the stdlib are unix-centric, Go having been designed and implemented by the original Unix and Inferno authors. But the FS interface is fairly generic and minimal which is great to encourage people to implement it, essentially a Stat and Open operations (for a read-only filesystem; I never really had a need for anything more as a lot can be implemented in terms of these 2 operations, e.g., glob). It is possible of course to nest this interface in a broader interface that supports writing and deletion.

[lerno]

@DrGo I guess permissions are also unix-centric to some extent, that seems to be part of this interface.

[DrGo]

you mean the FileInfo and FileMode types? I believe the new recommended approach is using DirEntry which does not include FileMode and as a bonus does not require a stat call.

[lerno]

Yeah, I was thinking about FileInfo, it did seem unix-centric.

[lerno]

I wonder what we should do about this proposal. Personally I don't see enough use-cases that we have to make something generic. And to make it generic – that is usually an OO thing anyway, where you thrive on plugin style coding. Should I reject this request? Does it fit somewhere? What would we use a generic, pluggable file system for? What functions would take some generic file system and do things to it?

[lerno]

If there is no further input I'll close it then?