opam broken on ubuntu 24.04 (`bwrap: operation not permitted`)

[dlesbre]

I've recently upgraded my computer from kubuntu 23.10 to kubuntu 24.04. This lead to errors with opam. Specifically, opam upgrade failed at trying to build the first package (dune.3.15.2), and opam switch create 4.14.1 also failed with the same error message:

bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted

Doing some digging, it seems to be related to ubuntu deciding to restrict unprivileged user namespaces https://ubuntu.com/blog/ubuntu-23-10-restricted-unprivileged-user-namespaces. I found a fix, detailed here: https://etbe.coker.com.au/2024/04/24/ubuntu-24-04-bubblewrap/ which got opam working again.

Config:

<include here the output of `opam config report`, if applicable>
# opam config report
# opam-version         2.1.5 
# self-upgrade         no
# system               arch=x86_64 os=linux os-distribution=ubuntu os-version=24.04
# solver               builtin-mccs+glpk
# install-criteria     -removed,-count[avoid-version,changed],-count[version-lag,request],-count[version-lag,changed],-count[missing-depexts,changed],-changed
# upgrade-criteria     -removed,-count[avoid-version,changed],-count[version-lag,solution],-count[missing-depexts,changed],-new
# jobs                 19
# repositories         2 (http) (default repo at 76de4586)
# pinned               1 (rsync)
# current-switch       4.14
# ocaml:native         true
# ocaml:native-tools   true
# ocaml:native-dynlink true
# ocaml:stubsdir       /home/dorian/.opam/4.14/lib/ocaml/stublibs:/home/dorian/.opam/4.14/lib/ocaml
# ocaml:preinstalled   false
# ocaml:compiler       4.14.1

[cuihtlauac]

UPDATE (Oct 15, 2024):

Thanks to @gegarcia, there is a safer way to fix this (see discussion below).

1. Install package apparmor-profiles
2. sudo ln -s /usr/share/apparmor/extra-profiles/bwrap-userns-restrict /etc/apparmor.d/
3. [Optional] If not using Ubuntu packaged Opam:
   1. cp /etc/apparmor.d/opam /etc/apparmor.d/local.opam
   2. In /etc/apparmor.d/local.opam replace /usr/bin/opam by the output of which opam
4. systemctl reload apparmor

It's working for me.

ORIGINAL POST, OBSOLETE (June 6, 2024):

Thanks @dlesbre, I've experienced the same issue. Upgrade to 24.04, and then opam upgrade failure. Doing as suggested by @etbe worked. I'm copying his suggestion here:

To resolve that you could upgrade to SE Linux, but the other option is to create a file named /etc/apparmor.d/bwrap with the following contents:

abi <abi/4.0>,
include <tunables/global>

profile bwrap /usr/bin/bwrap flags=(unconfined) {
  userns,

  # Site-specific additions and overrides. See local/README for details.
  include if exists <local/bwrap>
}

Then run systemctl reload apparmor.

[samvv]

I can confirm that the solution posted by @cuihtlauac worked. Might be worthwhile to add it to the docs?

[hhugo]

same here

[kit-ty-kate]

Reading the upstream issue https://bugs.launchpad.net/wike/+bug/2046844?comments=all i'm really not sure what we should do and i'm bit lost honestly. Adding a special message or a documentation telling the users to unconfine bubblewrap goes against the use of apparmor in the first place (to quote the dev): "Allowing Bubblewrap to do so would provide a loophole."

However looking at the apparmor sources, they seem to have a profile for opam already https://gitlab.com/apparmor/apparmor/-/commit/89a9f7673368e7d19802fd12ec3717a5477200e0 which was shipped in apparmor 4.0.0. Ubuntu 24.04 seems to have apparmor 4.0.1 so in theory it shouldn't be a problem anymore but if it is i'm not sure what was the point of the profile. Does anyone know anything about apparmor and are willing to work with upstream Ubuntu?

[etbe]

What you could do (and what I have done with SE Linux policy) is to have bubblewrap run with access to do such things and then transition back to another domain when it runs something.

Allowing a program to run another program with different levels of access potentially allows running another program with greater levels of access. But if we allow Bubblewrap to just do the things it needs to do (and not run random other programs with it's privileges) and then run another program in the original security context or a different restricted security context then it should be safe.

Forcing programs to be run with ONLY AppArmor and not Bubblewrap restrictions would not be a good thing. A common use of Bubblewrap is for a program to reduce the access of it's child processes which is something it will know about in detail and that AppArmor developers won't necessarily know. Even if AppArmor was configured to restrict child processes (which in many cases could be done) it wouldn't handle the case where an application developer changed their application design and needed matching changes to the restrictions on a child process.

[chris-armstrong]

However looking at the apparmor sources, they seem to have a profile for opam already https://gitlab.com/apparmor/apparmor/-/commit/89a9f7673368e7d19802fd12ec3717a5477200e0 which was shipped in apparmor 4.0.0. Ubuntu 24.04 seems to have apparmor 4.0.1 so in theory it shouldn't be a problem anymore but if it is i'm not sure what was the point of the profile. Does anyone know anything about apparmor and are willing to work with upstream Ubuntu?

Is it possible that this profile only works if opam is installed at /usr/bin/opam (probably the provided system package in Ubuntu) and not if it is installed using the binary distribution?

[WardBrian]

Which is the worse practice here, the above apparmor exception or disabling sandboxing?

[kit-ty-kate]

definitely disabling sandboxing

[gegarcia]

Please don't create an "unconfined" profile for bwrap. This completely bypasses the unprivileged user namespace creation restriction and makes your machine exploitable by attacks that use it.

You should instead create a profile for the application that uses bwrap instead. We have already created one for opam which is already shipped by the apparmor package.

AppArmor upstream has also created a profile for bwrap that does not make your machine vulnerable. You can find the profile here. It is not enabled by default on Ubuntu, but you can enable it by either copying the file from gitlab, installing it on /etc/apparmor.d/, and load it using sudo apparmor_parser /etc/apparmor.d/bwrap-userns-restrict.

You can also use the version shipped by the apparmor-profile package, and create a symbolic link from the extra-profiles directory to /etc/apparmor.d/ sudo ln -s /usr/share/apparmor/extra-profiles/bwrap-userns-restrict /etc/apparmor.d/ and load it sudo apparmor_parser /etc/apparmor.d/bwrap-userns-restrict.

[sanette]

Is it possible that this profile only works if opam is installed at /usr/bin/opam (probably the provided system package in Ubuntu) and not if it is installed using the binary distribution?

exactly. I had the same problem after installing opam on /usr/local/bin but trying again with /usr/bin, then it works! thanks!

[yawaramin]

So...can this issue be actioned in any way by opam maintainers? Or should it be closed?

[kit-ty-kate]

would handling the missing AppArmor profile in the opam install script be enough? This has the advantage of not requiring a fixed version pointing to some documentation to work:

bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/kit-ty-kate/opam/apparmor-install/shell/install.sh)"

I'm not sure what's best to do for people who install opam manually without the script though in the short term.

If that seems reasonable for you all, could you test this script and give me some feedback? I've tested it locally on the Ubuntu 24.04 qemu image and it worked fine. I'll open a PR later in the week if there is no pushbacks

[aogrcs]

Is it possible that this profile only works if opam is installed at /usr/bin/opam (probably the provided system package in Ubuntu) and not if it is installed using the binary distribution?

exactly. I had the same problem after installing opam on /usr/local/bin but trying again with /usr/bin, then it works! thanks!

This works for me, thanks

[mbarbin]

Is it possible that this profile only works if opam is installed at /usr/bin/opam (probably the provided system package in Ubuntu) and not if it is installed using the binary distribution?

Thanks that helped me find a workaround.

In case this helps someone else, my report is that I had this same issue apparently because I was using symbolic links to juggle different opam versions. For example:

/usr/bin/opam -> /usr/bin/opam-2.3.0-x86_64-linux

Replacing the symlink by an actual copy of the binary I'd like to use fixed it. I suspect whatever system config applies usually is defeated by a symlink textual expansion happening somewhere prior to its evaluation (this is a mere hypothesis though).

[kwj]

Just an FYI, in Ubuntu 25.04, the profile for bwrap bwrap-userns-restrict has been included in the apparmor package, not the apparmor-profiles package, so it seems to work fine with default settings.

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 25.04
Release:        25.04
Codename:       plucky
$ dpkg --list | grep apparmor
ii  apparmor                             4.1.0~beta5-0ubuntu14                      arm64        user-space parser utility for AppArmor
ii  libapparmor1:arm64                   4.1.0~beta5-0ubuntu14                      arm64        changehat AppArmor library
$ dpkg -S /etc/apparmor.d/bwrap-userns-restrict
apparmor: /etc/apparmor.d/bwrap-userns-restrict
$ sysctl -a 2>/dev/null | grep unpriv
kernel.apparmor_restrict_unprivileged_unconfined = 1
kernel.apparmor_restrict_unprivileged_userns = 1
kernel.unprivileged_bpf_disabled = 2
kernel.unprivileged_userns_clone = 1
net.ipv4.ip_unprivileged_port_start = 1024
vm.unprivileged_userfaultfd = 0
$
$ opam --version
2.3.0
$ opam update

<><> Updating package repositories ><><><><><><><><><><><><><><><><><><><><><><>
[default] no changes from https://opam.ocaml.org
$ opam switch list
#  switch   compiler                                           description
→  5.3.0    ocaml-base-compiler.5.3.0,ocaml-options-vanilla.1  ocaml-base-compiler = 5.3.0 | ocaml-system = 5.3.0
   default  ocaml.4.14.1                                       default
$ opam switch create 5.2.1

<><> Installing new switch packages <><><><><><><><><><><><><><><><><><><><><><>
Switch invariant: ["ocaml-base-compiler" {= "5.2.1"} | "ocaml-system" {= "5.2.1"}]

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
⬇ retrieved ocaml-config.3  (cached)
∗ installed base-bigarray.base
∗ installed base-threads.base
∗ installed base-unix.base
∗ installed ocaml-options-vanilla.1
⬇ retrieved ocaml-base-compiler.5.2.1  (cached)
∗ installed ocaml-base-compiler.5.2.1
∗ installed ocaml-config.3
∗ installed ocaml.5.2.1
∗ installed base-domains.base
∗ installed base-nnp.base
Done.
# To update the current shell environment, run: eval $(opam env --switch=5.2.1)
$ opam switch list
#  switch   compiler                                           description
→  5.2.1    ocaml-base-compiler.5.2.1,ocaml-options-vanilla.1  ocaml-base-compiler = 5.2.1 | ocaml-system = 5.2.1
   5.3.0    ocaml-base-compiler.5.3.0,ocaml-options-vanilla.1  ocaml-base-compiler = 5.3.0 | ocaml-system = 5.3.0
   default  ocaml.4.14.1                                       default

[WARNING] The environment is not in sync with the current switch.
          You should run: eval $(opam env)
$ 

[kit-ty-kate]

@kwj Is that profile really loaded by default? Where is your opam binary located at? (command -v opam).

When the opam binary is installed in /usr/bin there wasn't ever any problem. If it's installed in /usr/local/bin or anywhere else however there should still be issues with it unless they're actually loading the bwrap profile or changed the opam profile, as far as i understand it.

[kwj]

I installed the opam binary outside of /usr/bin and have used it. Also, nothing has been changed in the profile.

$ command -v opam
/opt/opam/c/bin/opam
$ realpath /opt/opam/c/bin/opam
/opt/opam/2.3.0/bin/opam
$ ls -l /opt/opam/2.3.0/bin/opam
-rwxr-xr-x 1 root root 8824344 Nov 27 16:44 /opt/opam/2.3.0/bin/opam
$ 
$ grep opam /etc/apparmor.d/* 2>/dev/null
/etc/apparmor.d/opam:profile opam /usr/bin/opam flags=(unconfined) {
/etc/apparmor.d/opam:  include if exists <local/opam>
$ cat /etc/apparmor.d/opam
# This profile allows everything and only exists to give the
# application a name instead of having the label "unconfined"

abi <abi/4.0>,
include <tunables/global>

profile opam /usr/bin/opam flags=(unconfined) {
  userns,

  # Site-specific additions and overrides. See local/README for details.
  include if exists <local/opam>
}
$ 

Until now, I bypassed this restriction with kernel.apparmor_restrict_unprivileged_userns = 0.

[kit-ty-kate]

ah, good to know it's been fixed upstream then. Thanks for the update!

[gm0stache]

would handling the missing AppArmor profile in the opam install script be enough? This has the advantage of not requiring a fixed version pointing to some documentation to work:

bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/kit-ty-kate/opam/apparmor-install/shell/install.sh)"

I'm not sure what's best to do for people who install opam manually without the script though in the short term.

If that seems reasonable for you all, could you test this script and give me some feedback? I've tested it locally on the Ubuntu 24.04 qemu image and it worked fine. I'll open a PR later in the week if there is no pushbacks

thanks for providing the script! fixed the issues for me (on Ubuntu 24.04.2 LTS).