Mountable file system

[shabbyrobe]

Here's a feature that we are using on my current project - a mountable file system, similar to unix bind mounts. I thought I should polish it up and submit it back in case others find it useful.

Basic usage:

base := afero.NewBasePathFs(afero.NewOsFs(), "/tmp")
fs := afero.NewMountableFs(base)

child := afero.NewMemMapFs()
_ = fs.Mount("/child", child)

in := []byte("1")
afero.WriteFile(fs, "/child/test", in, 0644)
out, _ := afero.ReadFile(child, "/test")

// should print true
fmt.Println(bytes.Equal(in, out))

Currently the tests have it nearly 80% covered. I thought I'd hold off polishing it any further in case there was feedback that needed to be addressed.

I had a few questions about how best to organise things - I notice that some filesystems are in the root afero package, but others have their own subpackages. This exports a fair few types, should I move it into a package of its own?

I have written a few tests against OsFs which make use of a temp directory which is destroyed in a defer. I've tried to write as few tests as possible in that way as I wasn't sure if this was ok to do but some of them really benefited from hitting the OsFs implementation as well.

[CLAassistant]

All committers have signed the CLA.

[shabbyrobe]

Hmm, looks like I have some work left to do on Windows. Will sort it out tomorrow at the office when I get access to a Windows box.

[xor-gate]

This looks very interesting!

[shabbyrobe]

Some things are still NQR - I notice that some of the other filesystems in Afero are protected with Mutexes, I'm figuring this one probably should be too.

I also reckon I'm not quite done finding bugs with the intermediate nodes that it uses when you mount in a nested dir that does not exist. I tried to do it the linux way of requiring a directory to exist in order to mount over it, but that code ended up being at least as much of a mess if not more, and less convenient.

This also puts a lot of extra types in the afero package, whereas MemFS has split itself up a bit and put its own tools in the afero/mem package. Maybe this one should do the same?

[shabbyrobe]

Please pardon the bump; I was wondering if there was any chance of this being accepted, and if there's anything anyone would like me to do to improve it.

[object88]

I'd like to echo this bump; this looks just about perfect for my current project.

[shabbyrobe]

I'm happy to do whatever needs to be done to get it up to scratch for inclusion.

One of the things I'm still not sure about is if there is a core policy on whether things need to be protected by a mutex. Only the memmapfs seems to have one; should this?

[object88]

So, I played with this some more today, and I found that I could do what I needed using the existing functionality. I don't say this to discourage your work or to say that this PR is unnecessary, but just in case someone else is looking for similar functionality.

My need is to use the OS file system, plus some a memory-backed temporary file system on top. I was able to use afero.CopyOnWriteFs. This paraphrases my code:

func createOverlay(files map[string]string) (string, afero.Fs) {
	fs := afero.NewMemMapFs()
	rootPath := filepath.Join(string(os.PathSeparator), uuid.New().String(), "go", "src")

	for path, contents := range files {
		completePath := filepath.Join(rootPath, path)

		fs.MkdirAll(filepath.Dir(completePath), 0644)
		fh, _ := fs.Create(completePath)
		fh.WriteString(contents)
		fh.Close()
	}

	return rootPath, fs
}

func Test_Foo(t *testing.T) {
	src := `package bar
	import "fmt"
	var BarVal string = fmt.Sprintf("0")`

	rootPath, overlayFs := createOverlay(map[string]string{
		filepath.Join("bar", "bar.go"): src,
	})

	// ...

	l1 := NewLoader(le, barPath, "darwin", "x86", runtime.GOROOT(), func(l *Loader) {
		l.fs = afero.NewCopyOnWriteFs(l.fs, overlayFs)
	})

	// ...

}

In this case, l.fs is normally just afero.NewReadOnlyFs(afero.NewOsFs()), and this code appends a memory-backed file system on top at a randomly-generated path off the root.

[object88]

And now I'm back to "I'd like this please," as I have a new scenario that I can't figure out how to handle. Because $REASONS, I want to mount a subsection of a real file system into my view of the file system. As an example:

base := afero.NewOsFs()
fs := afero.NewMountableFs(base)

bfs := afero.NewBasePathFs(afero.NewOsFs(), "/base/path")
_ = fs.Mount("/new/and/improved/path", bfs)

// I can now access both `/base/path/foo.txt` and `/new/and/improved/path/foo.txt`
// for the same file.

[shabbyrobe]

Hmm, it appears my git user name was different for some of the commits, and I can't see a way to make CLA assistant accept that user. Sorry for the inconvenience; how do I certify that "Blake Williams" == "shabbyrobe"?

[mankins]

Hmm, it appears my git user name was different for some of the commits, and I can't see a way to make CLA assistant accept that user. Sorry for the inconvenience; how do I certify that "Blake Williams" == "shabbyrobe"?

@shabbyrobe maybe you need to link the commits?

[chaoqing]

@shabbyrobe, Interesting, wish I came cross this fork early. I implemented a similar functionality https://github.com/chaoqing/afero/blob/e47db790683a213a35daa9d222a611cda5452204/rclonefs/bind.go#L53

I may switch to this fork after reviewing the code.