OverlayFS support

[tailhook]

This is a big ticket tracking overlayfs support.

The plan, in the order of importance:

• [x] Check whether unprivileged overlayfs is supported (vagga _check_overlayfs_support)
• [ ] Use overlayfs for !Container step. This will be enabled by a container-level setting. Which, presumably, will always be disabled by default.
• [ ] Make of use of overlayfs for Snapshot volume. This will be enabled by a volume-level setting initially. In perspective the setting will be default on systems that support it
• [ ] !Remove / clean_dir could probably be optimized for the layered fs. I.e. should be able to remove a directory without traversing contents
• [ ] Use overlayfs for _run --writable and transient copies
• [ ] Add vagga _build container --cache-each-step to ease debugging of container builds (actually to be able to continue failing build from any failed step)

I think we should try at least to support !Container for the next milestone (which will be 0.7.0)

[piranna]

Would this allow to mount an overlayfs filesystem from inside a running vagga instance?

[tailhook]

@piranna you already can do this, if you have the kernel which supports unprivileged overlayfs (ubuntu has, as well as our arch linux kernel build). You can check it with vagga _check_overlayfs_support. But otherwise it has nothing to do with vagga using the overlayfs support per se.

[piranna]

I supposed, just wanted to be sure because last time I tried to do it I was not able, and yesterday trying to do the same with Docker my machine crashed each time I tried it. Native mounting on QEmu or real hardware (no containers) works flawlessly, so probably there's a problem in other place.

[jirutka]

Is it really a good idea to use OverlayFS for layered FS? Based on some sources I read, it’s far from stable, it’s one of many mistakes of Docker Inc…

I think that it’d be better to utilize Btrfs or LVM for this feature.

[tailhook]

@jirutka, can you share a link? I though docker had problems with first version of overlayfs driver, but they have fixed all issues in the second version (by "overlayfs driver" I mean some code in docker that handles overlayfs, they have same in-kernel subsystem).

I think that it’d be better to utilize Btrfs or LVM for this feature.

We can't use LVM as unprivileged user. Also, LVM is quite wasteful (because it copies on write whole disk blocks which usually are ~ 10MiB), up to the level where it doesn't make sense to use (i.e. doesn't provide any savings comparing to full copy).

Btrfs support is planned. But there are not many systems which use btrfs for their /home directory. And setting up storage-dir with btrfs filesystem is too complex for most users (for example most home computers and workstations have a maximum-sized /home, so can't create another partition easily).

On the other hand, we can achieve quite similar savings by just using hardlinks. It's a lot less safer, and doesn't work for all use cases, but can provide as good results as using btrfs for many simpler use cases (but will not work for Snashot volumes, for example).

[jirutka]

can you share a link?

For example this one: https://thehftguy.com/2016/11/01/docker-in-production-an-history-of-failure/. And my co-worker warned me about using OverlayFS on ext4, but don’t remember details.

On the other hand, we can achieve quite similar savings by just using hardlinks.

How? Hardlinks can work only if you can ensure that files will not be modified in-place, but always recreated.

[tailhook]

How? Hardlinks can work only if you can ensure that files will not be modified in-place, but always recreated.

Sure. We can hardlink equal files after image has been built, to save space. As I've already said it doesn't help with Snapshot volumes. But for all other purposes we mount container image read-only when running all the commands.

[jirutka]

We can hardlink equal files after image has been built, to save space.

Aha, sounds like a good solution!