GPT + FAT32 Disk support

[fintelia]

[celskeggs]

I've made some progress on cleaning up our disk handling systems:

• Making bus locations usable for disk selection
• Eliminating the legacy ideread/idewrite preprocessor-controlled interface
• Getting the IDE driver to work again

This should make it possible to treat disk partitions just like bare disks. I'll be working on MBR and/or GPT partition support on Monday.

It also appears to be the case that our filesystem is currently configured in a way where it won't actually write any changes to disk; there may be clean-up of the fs.cc and mfs.cc code necessary before I can add FAT32 support -- but I'll be able to test partitions without that.

[celskeggs]

MBR and GPT support is in; I'm in the process of testing it on bhw2. I discovered that the filesystem layers are tightly integrated into the entire codebase, so I'm working on adding an abstraction layer so that I can add FAT32 support afterward.

[fintelia]

If it would be signficantly easier, we could also consider supporting just FAT32 instead of our own bespoke filesystem format.

[celskeggs]

The bespoke filesystem is very tied into the guts of sv6, so I don't think it'll be much easier to do that.

[celskeggs]

Almost done cleaning up the filesystem interfaces; once I finish removing the dependency on the filesystem for MAP_SHARED mappings, I should be able to start implementing FAT32.

[celskeggs]

MAP_SHARED is cleaned up!

I realized that it's going to be very difficult to debug a FAT32 filesystem when all of our userspace tools are stored in said filesystem. So I'm taking a bit to setting up a very simple virtual filesystem. I have a version that works for the "only a root directory is mounted" situation, and appears to be stable -- I have some datastructures whose implementation I messed up, so it doesn't actually let me mount additional filesystems. Once those are fixed, I'll be on to the FAT32 implementation.

[celskeggs]

Got the VFS implementation to a working place and started on FAT32 support.

Current progress on FAT32 is in 9b2b72898c70fd7374ad01e640a9e2a4febfa0dd; it is currently a preliminary implementation that is both broken and incomplete. However, once I finish debugging it, it should be a functional (albeit non-threadsafe and read-only) FAT32 driver, and then I can iterate from there with better data structures, better algorithms, less unsafety, and more features (like actually being able to write files).

[celskeggs]

FAT32 is enabled on master and can now execute programs. It does not yet support writing. Next steps:

• [ ] Making the filesystem writable:
  • [x] set up lookup table of vnodes
  • [x] set up cache for directory children
  • [ ] modifying file metadata in directories
  • [x] truncating files
  • [x] writing files
  • [x] writing out dirty cached clusters to disk
  • [x] creating regular files
  • [x] creating files with long filenames
  • [x] creating directories
  • [ ] creating devices
  • [ ] creating sockets
  • [x] removing files
  • [x] removing directories
  • [ ] renaming files
  • [ ] mounting over FAT32 folders
  • [ ] make sure that vnodes can actually be garbage-collected
  • [x] preventing modification unless a special "write enable" file is found
  • [ ] updating modification, creation, and access times
• [x] Making the filesystem case-insensitive to match linux support
• [x] Adding long filename support
• [ ] Checking for concurrency bugs
• [ ] Fix all known concurrency bugs (see FIXMEs, TODOs)
• [ ] Gracefully handle OOM condition in FAT32 buffer cache when entries cannot be evicted
• [ ] Lazily populate cluster ID lists for vnodes (i.e. currently listing a directory does this for every entry in the directory, unnecessarily)
• [ ] Figure out how filename capitalization should be handled

[celskeggs]

Very limited writable filesystem support is now in master; I can truncate, extend, append, and modify files, and write back the changed sectors to disk.

I've been keeping a checklist in the above comment updated as I go.

While I've built a lot of the plumbing required to modify data in the filesystem, the plumbing required to change directory entries is currently limited to changing file sizes, which is why quite so many features above are incomplete, despite the work I've done so far.

[celskeggs]

A bunch more support is in; latest part is directory creation. See above checklist for progress.

I'm going to leave this as-is for now; it should be sufficient to transfer data between systems on real disks (though I have not tested it on real disks yet). It is not functional as a root file system, and is still missing some essential features -- like renaming files.