docs: discourage `/tmp/lima` on shared hosts (and deprecate it?)

[AkihiroSuda]

Using /tmp/lima is discouraged on shared hosts, as another user on the host can peek/poke the directory.

Maybe we should deprecate and remove it in v2.0 ? A user can still enable the directory by specifying --mount /tmp/lima:w.

[jandubois]

Using /tmp/lima is discouraged on shared hosts, as another user on the host can peek/poke the directory.

Only users with the same GID can poke because we create the directory with 750 permissions. Maybe we should create non-existing directories with 700 instead?

Since /tmp normally has the sticky bit set, only the owner or root can change the permission flags, even if the user otherwise has write access to the directory itself.

[jandubois]

SSH agent forwarding puts the agent socket in a directory on /tmp, relying on file mode protection bits being sticky. If it is good enough for that use case, then it should be good enough for /tmp/lima.

[AkihiroSuda]

Maybe we should create non-existing directories with 700 instead?

Yes, but TOCTOU still has to be cared carefully

BTW, in the current design multiple users cannot run Lima simultaneously by default, so it should be renamed to /tmp/lima-{{.Username}}?

[jandubois]

BTW, in the current design multiple users cannot run Lima simultaneously by default, so it should be renamed to /tmp/lima-{{.Username}}?

We currently use {{.GlobalTempDir}} in our default mounts:

mounts:
- location: "{{.GlobalTempDir}}/lima"
  mountPoint: /tmp/lima
  writable: true

We talked about using {{.TempDir}} whenever we are willing to make an incompatible change (Lima 2.0 release). It will use the value of $TMPDIR instead, so should be a private directory.

We could add additional code that checks if the return value is still /tmp, and if it is, we check if /run/user/$UID exists, and use that if it does.

It would only affect Linux, as macOS always has $TMPDIR pointing to a private temp directory.

Not sure I like it though, a bit too much magic, but what do you think?

[jandubois]

Maybe we should create non-existing directories with 700 instead?

It looks like this is currently not very consistent. We seem to create directories for forwarded sockets with 750 permissions. Mount locations are created with 755 when using reverse-sshfs and 750 when using virtiofs.

I would suggest to change all of them to 700. This only applies to directories created by Lima. If they already exist, then their existing permissions should not be changed.

[afbjorklund]

This was one of the reasons for making /tmp/lima into a variable:

• https://github.com/lima-vm/lima/issues/2339

The other reason was that Fedora puts /tmp in RAM so the space is limited...

If you are doing something like building images, might want to use /var/tmp

[thomasjm]

BTW, in the current design multiple users cannot run Lima simultaneously by default, so it should be renamed to /tmp/lima-{{.Username}} ?

We talked about using {{.TempDir}} whenever we are willing to make an incompatible change (Lima 2.0 release). It will use the value of $TMPDIR instead, so should be a private directory.

Either of these ideas would improve the story for multiple users on the same machine using Lima. But both of them would still cause different Lima instances belonging to the same user (or login session) to share the same /tmp dir. It doesn't make sense to me why separate VMs would share a /tmp dir.

Possible dumb question, but why do the default Lima templates mount "/tmp" from the host at all? It seems like the cleanest option would be to not do this.

[AkihiroSuda]

$TMPDIR

The path is long, ugly, and hard to locate from a terminal

Possible dumb question, but why do the default Lima templates mount "/tmp" from the host at all? It seems like the cleanest option would be to not do this.

Because I wanted to have a writable directory while keeping the home read-only by default

[thomasjm]

Because I wanted to have a writable directory while keeping the home read-only by default

Gotcha -- is that something that's important for the operation of Lima somehow (or some of the templates)?

To me it's somewhat surprising as a user that Lima auto-mounts anything at all, and I always have to go digging through the docs to remind myself how it works. As far as the docs go, it's not mentioned on the filesystem mounts page; the only reference I can find is under examples.

[jandubois]

Because I wanted to have a writable directory while keeping the home read-only by default

I always mount the home directory writable, so I rarely ever need the temp mount.

I think we should consider making it opt-in instead of opt-out in Lima 2.0.

A simple way to do this would be to move it to template://_default/temp-mount.yaml.

I've already been contemplating if we should allow specifying additional base templates on the commandline (they would be relative to the first template if using a relative path), so you could write:

limactl create -y template://ubuntu _default/temp-mount.yaml

and it would be the same as if the template was

base:
- ubuntu.yaml
- _default/mounts.yaml
- _default/temp-mount.yaml

Of course this is still somewhat clunky, and I'm hoping that we could use something like --param temp-mounts instead, but that depends on how soon we want to have a Lima 2.0 release.

Or we could add a default:// URL scheme, so default://temp-mount would be shorthand for template://_default/temp-mount. It also hides the somewhat ugly implementation details of the _default subdirectory from the user:

limactl create -y template://ubuntu default://temp-mount

Anyways, lots of bikesheds that can be painted...

[thomasjm]

I've already been contemplating if we should allow specifying additional base templates on the commandline

I like the idea of increasing the configurability. But doing it in terms of templates would require the user to dig into and understand how the templates work. Which to me would mean either looking in the Lima repository or finding the templates dir where it's installed on my system, neither of which is too appealing.

Lima already has the --mount argument, which IMO is the right level of granularity for this. Making it even more similar to Docker's --volume argument for bind-mounting would be even better :)

I mean, consider the clarity of something like this (inventing some syntax a bit):

limactl create -y template://ubuntu --mount "$(mktemp -d):/tmp:w"

[jandubois]

Which to me would mean either looking in the Lima repository or finding the templates dir where it's installed on my system

Regardless of this current discussion, I think the documentation for templates need some serious improvements. So at least for all bundled templates it needs to contain the information how they can be further customized / parameterized.

I mean, consider the clarity of something like this (inventing some syntax a bit):

limactl create -y template://ubuntu --mount "$(mktemp -d):/tmp:w"

I like it, and should be easy to implement.

[AkihiroSuda]

limactl create -y template://ubuntu --mount "$(mktemp -d):/tmp:w"

This already works

[afbjorklund]

To me it's somewhat surprising as a user that Lima auto-mounts anything at all, and I always have to go digging through the docs to remind myself how it works.

This should at least be mentioned, not all users are coming from a Docker Machine and expecting all of their home directory to be exported to the VM (with the /tmp/lima being the writable workaround for the read-only default mount in 1.0)

For instance we have some ex-Vagrant users, that are expecting the VM to use a unique user name (like "vagrant" or "lima") and only export the current working directory ($PWD) under some unique location (like "/vagrant") and not the home.

---

There were also some similar "complaints" about containerd being installed by default, but not hard to avoid that either.

--containerd none

[jandubois]

limactl create -y template://ubuntu --mount "$(mktemp -d):/tmp:w"

This already works

I don't think so. --mount /foo:w works, but --mount /foo:/bar:w doesn't. You can specify the location, but not the mountPoint. Is easy to fix though: if the argument has 2 : in it, then the middle term is the mountPoint.

[jandubois]

For instance we have some ex-Vagrant users, that are expecting the VM to use a unique user name (like "vagrant" or "lima") and only export the current working directory ($PWD) under some unique location (like "/vagrant") and not the home.

Not every use case can be the default, unless they are all the same use case. The history of the Lima project is to make it easy to run containers on macOS and pretend they are running locally and not in a VM.

We have worked quite a bit to make Lima more generally useful and configurable, but that doesn't mean we have to abandon the original use case. Note that "make it easy" was not accidental, but a design goal.

You can always use --plain if you don't want any of the Lima features that integrate the VM with the host. Or you can easily create a template for whatever you want now:

base: template://_images/ubuntu

containerd:
  system: false
  user: false

user:
  name: vagrant
  uid: 1000
  home: /home/vagrant

We do not support referencing $PWD or other environment variables in the template. It is not clear how they should work: do they need to be interpolated when you create the instance, or when you start it. Should limactl restart change the mount locations because you are in a different directory now?

[msimkunas]

Slightly off-topic but why is the user's home directory automounted? I was a little taken aback by this. I'd rather Lima used a secure-out-of-the-box approach and would not expose the contents of $HOME to the VM by default.

Of course, it's the user's fault if they run an insecure template but the default configuration should at the very least attempt to contain the potential damage. Exposing the entirety of $HOME to the VM seems completely unnecessary.

In most cases, this may not even be desired because you will usually want to mount a directory that contains your projects only anyway.

[AkihiroSuda]

Slightly off-topic but why is the user's home directory automounted?

For allowing (docker|podman|nerdctl) run -v $(pwd):/mnt to just work, as in Docker Desktop.

I'd rather Lima used a secure-out-of-the-box approach and would not expose the contents of $HOME to the VM by default.

You can use limactl start --mount-none to disable all the mounts.

Exposing the entirety of $HOME to the VM seems completely unnecessary.

We could intercept invocation of (docker|podman|nerdctl) run -v and make mounts dynamically, but it could be quite complicated and less robust.

In most cases, this may not even be desired because you will usually want to mount a directory that contains your projects only anyway.

Lima v2.0 is planned to have limactl shell --sync-host-workdir to implement that (but rsync instead of mount, probably)

• https://github.com/lima-vm/lima/issues/3711

$ limactl start --mount-none template://claude
$ cd ~/some-project
$ limactl shell --sync-host-workdir claude claude
Changed files:
- foo.go
- Makefile

⚠️Accept the changes?
→ Yes
  No
  View the changed contents

[msimkunas]

@AkihiroSuda Thanks for clearing it up. I've also found that you can basically override everything via a template, so that works as well.

[afbjorklund]

Lima already has the --mount argument, which IMO is the right level of granularity for this. Making it even more similar to Docker's --volume argument for bind-mounting would be even better :)

I used the same option (--volume) for Podman Machine, and it has been confusing users ever since. For some reason they mix it up with Podman Engine volumes and bind mounts, which are different...

So I think having two different names for it is probably a good idea. Even if it's two different projects. But the mountpoint syntax should be possible to add, if ignoring the name of the flag for a second.