shiftfs incompatible with overlayfs without specific overlayfs version

[hxx3000]

There seems to be an additional kernel patch in conjunction with shiftfs when used above overlayfs:

https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/focal/commit/fs/overlayfs?id=d24b8a547be1578cb5a200ad9ac57a258f0a9de1

So it should be noted in the documentation not to use this dkms in conjunction with overlayfs.

[toby63]

For everyone reading this issue I want to give a summary:

Note: The following information might be outdated.

Problem: The problem is that overlayfs cannot be used inside a container, when the container software uses shiftfs. Note: This is only a problem for special use cases, e.g. running Docker with the overlayfs-storage-driver inside an LXD container.

Solution: A solution is available in form of kernel patches for the overlayfs module (see ubuntu kernel commit). But these patches are not included in the mainline kernel. To my knowledge only Ubuntu included them in their kernel.

Main (potential) Workaround:

• patch the kernel module "overlayfs" via dkms:
  (or more correct: install a modified/patched version of overlayfs via dkms)
  You can use a similar dkms package (such as this one) and try to edit it for overlayfs. Note: This approach is untested, so it might not work (e.g. some modules cannot be "managed" with dkms).

Other Workarounds:

• use Ubuntu: Regarding Derivates: it is unknown to me whether Ubuntu derivates (such as mint) include the ubuntu kernel

• convince your distro's kernel maintainers to include the patches

• (not recommended) patch the kernel sources and rebuild the kernel manually (this might take a long time to compile)

---

Original Post: @hxx-fetch: You are right. Although this problem is mentioned in a post I linked to in the Readme (see "Further information on shiftfs"), I agree that I should clarify it here in the Readme too.
Quote from the LXD Forum:

Limitations

The only limitation we’re currently aware of is that shiftfs will prevent the use of overlayfs inside the container. This may break Docker users. If this applies to you, you should stay away from shiftfs for now.

I added a note in the Readme (see commit).

[toby63]

I was not aware of a patch for this though, interesting :thinking: .

@brauner: Some questions about this:

1. Just for clarity: This problem only occurs inside containers? So using overlayfs on the host still works (e.g. using docker on the host)?

2. Will the patch(es) be in the mainline kernel? If yes, what is the status of implementation?

3. I assume it is not possible or "easy" to put the patch(es) into a kernel module?

[brauner]

I was not aware of a patch for this though, interesting .

@brauner: Some questions about this:

1. Just for clarity: This problem only occurs inside containers? So using overlayfs on the host still works (e.g. using docker on the host)?

Yes.

2. Will the patch(es) be in the mainline kernel? If yes, what is the status of implementation?

We're working on an upstreamable version of shiftfs but it'll be a little while.

3. I assume it is not possible or "easy" to put the patch(es) into a kernel module?

Not impossible but unlikely to be accepted.

[toby63]

@brauner: Thank you for the information.

3. I assume it is not possible or "easy" to put the patch(es) into a kernel module?

Not impossible but unlikely to be accepted.

Sry for asking, but that means: a) accepted by the kernel maintainers? b) accepted by the system/kernel (for example my system)?

And it would require a rewrite I assume?

I was just thinking of implementing that into a dkms skript once again :smile: .

[brauner]

@brauner: Thank you for the information.

1. I assume it is not possible or "easy" to put the patch(es) into a kernel module?

Not impossible but unlikely to be accepted.

Sry for asking, but that means: a) accepted by the kernel maintainers?

Yes, unlikely to be accepted upstream because we're (as in us, the kernel maintainers) how exactly we want this feature to look upstream.

b) accepted by the system/kernel (for example my system)?

No, I meant a). I don't know what your maintainers would say. :)

And it would require a rewrite I assume?

I was just thinking of implementing that into a dkms skript once again .

If you can find a way for that patch to be included in the dkms script, then sure. :)

[toby63]

@brauner:

If you can find a way for that patch to be included in the dkms script, then sure. :)

1. Can dkms alter existing files in the kernel? 1.a) Or replace existing files or modules? I found inconsistent answers so far, but some suggest it is possible (but only for modules). A potencial way of doing so, is in comment 7.

2. Or should I find a workaround like including the patch lines in some other way (e.g. new files)? Imo not possible without rewrite, because programs are using overlayfs, so overlayfs has to include the patches.

3. ~~Are these two (a b) the only necessary commits?~~ Edit: I see now; so commit d24b8a547be1 is a replacement of commit 37430e430a14 ? And that commit/patch is the only one needed?

[toby63]

Found a potencial solution to patch overlayfs via dkms: https://www.reddit.com/r/linux/comments/6g4l7n/using_dkms_to_compile_patched_modules/

Update: I am working on a bit different solution. Reasons for this include:

• The method from the link uses kernel-source packages (which are quite huge (~800 MB))
• Some changes are necessary

If this works, the solution will likely be a seperate dkms script for patching overlayfs.

[toby63]

@brauner: I also found a problem (applies only for my repo, so no worries):

Your commit includes a check for SHIFTFS_MAGIC:

#fs/overlayfs/super.c:
if (!err) {
			if ((path->dentry->d_sb->s_magic != SHIFTFS_MAGIC) && ovl_dentry_remote(path->dentry)) {

In my dkms script for shiftfs I did not add the magic number to "include/uapi/linux/magic.h" (maybe not possible with dkms?). Instead I used a workaround suggested by stgraber: I only added the magic number to the Makefile of shiftfs (via EXTRA_CFLAGS).

So I guess overlayfs cannot read the magic number. I would solve this by also adding the magic number via EXTRA_CFLAGS to the Makefile of overlayfs, or would you suggest a different solution?

[toby63]

@hxx-fetch Sry to bother, am I assuming right that you use the docker setup this problem is about? (just as a reminder: use of docker with overlayfs inside lxd container)
If yes; I am testing the docker setup (with my workaround, see comment 2) now, and somehow it does not work. Now I want to rule out certain setup mistakes.

So it would be nice if you could answer me some questions:

• what storage backend do you use for lxd?
• what image (or OS) do you use for the lxd container?
• What version of docker do you use (inside the container of course)?
• Do you use the overlay2 storage driver in docker?
• did you use any other special options for lxd or docker that could be relevant for this setup?

Also if someone else is reading this, you can also answer :wink: . For convenience, the relevant info commands are:

• lxc info
• lxc config show containername
• docker info

[hxx3000]

@toby63 Hi,

Sorry, I don't think I'm much of help with this issue:

I was looking for a solution how to get shiftfs run on gentoo-linux, not wanting to manage custom kernel patches by myself. I then found your project and seeing the overlayfs-patches were not included, I thought I give you a notification so docker/overlayfs users won't have unexpected problems.

So what I have currently running is lxd 3.16 on gentoo-linux using zfs, but no docker/overlayfs. The containers are debian, ubuntu and gentoo I plan to use your shiftfs dkms when lxd 4.0 will be avail in gentoo.

[toby63]

@hxx-fetch Thank you for answering anyway :slightly_smiling_face: . Also glad my project is useful for you & thank you for the issue report, because I was aware of this problem, but I haven't mentioned it here and I also was not aware of available patches.

In general (for everyone reading this): ~~I have one idea left to test, but if that does not run, I am out of clues :smile: .~~ But honestly this problem seems to be veeery special to me more and more. Because the standard storage backends for lxd (btrfs or zfs (at least thats the recommendation)) and docker (overlayfs) are not compatible (at least thats my impression).
So docker will either use btrfs or zfs as well or you need to use a different backend for lxd (dir with ext4 on hosts partition; which is what I try to test).

Update: I could not get it to work, but that might be a problem of my setup etc., so if you want to try my scripts yourself, just write a comment below :wink: .