More Robust Filesystem Manipulation

[NathanTP]

Mounting target images is fraught. It's the source of many bugs (e.g. #168 #137). It also prevents CI for marshal (since Docker doesn't support mount or FUSE). We should really deal with these issues at the root rather than patching each issue with hacks.

I think the most promising method is to use e2fsprogs to manually copy files in and out of the filesystem. It would require some changes to the build process (that uses mounts to simplify data transfers), but it could be done.

[timsnyder-siv]

Along the lines of using e2fsprogs and issues with mounting filesystems, have you done anything to make sure that the filesystem images are bit identical each time you build them with firemarshal? Obvi, there will be impact from the bundled systems you're using to build the images (buildroot, whatever else) but do you or could you do something to normalize them to meet the constraints for repeatability placed on you by FireSim (bit exactness)?

One thing that we've found helpful because we also manipulate the images a bit downstream from firemarshal so that we can have a single image for SPEC and tweak the init scripts to run specific workloads, is to run the rootfs through make_ext4fs -T <fixed_timestamp> and that makes all of the timestamps always whatever time we pick. Picking 0 is fraught because some things (ahem runspec) are clever and don't like it if you run them before they think they sprang into existence.

I've been looking for an equivalent way in e2fsprogs to set a fixed timestamp on all files in the created filesystem, but I don't see one (yet).

Also, if you'd like a pointer to FUSE working in CI, take a look at https://github.com/sifive/wake/tree/c91b5c45dfa7e9881c708a1673500821ce671976/.circleci. It has all the right knobs twiddled to enable FUSE (at least on CircleCI).