google sandboxed api fails in docker

[tsosea]

To reproduce:

Create and run a docker container with ubuntu 18.04 (also tried running with --privileged). Then, follow the quick start from the sandboxed api main page https://developers.google.com/sandboxed-api/docs/overview. The run fails at the following command (from the quick start):

bazel run //sandboxed_api/examples/stringop:main_stringop

The error is:

`[==========] Running 6 tests from 1 test suite. [----------] Global test environment set-up. [----------] 6 tests from StringopTest [ RUN ] StringopTest.ProtobufStringDuplication [sandboxed_api/sandbox2/util.cc : 138] RAW: clone(): Operation not permitted [1] [sandboxed_api/sandbox2/forkserver.cc : 559] RAW: Check (pid != -1) failed: failed to fork initial namespaces process: Operation not permitted [1] [sandboxed_api/sandbox2/forkserver.cc : 219] RAW: Receiving init PID from the ForkServer failed WARNING: Logging before InitGoogleLogging() is written to STDERR E20200710 03:08:35.924361 4259 executor.cc:162] Could not obtain init PID [sandboxed_api/sandbox2/comms.cc : 535] RAW: write: Bad file descriptor [9] [sandboxed_api/sandbox2/forkserver.cc : 188] RAW: Sending PB to the ForkServer failed E20200710 03:08:35.925424 4260 executor.cc:162] Could not obtain init PID E20200710 03:08:35.925566 4247 transaction.cc:61] Tried 2 times to run the transaction, but it failed. SAPI error: 'UNAVAILABLE: Could not start the sandbox'. Latest sandbox error: 'SETUP_ERROR - Code: FAILED_SUBPROCESS' sandboxed_api/examples/stringop/main_stringop.cc:61: Failure Value of: st.Run([](sapi::Sandbox* sandbox) -> absl::Status { StringopApi api(sandbox); stringop::StringDuplication proto; proto.set_input("Hello"); sapi:✌️:Protostringop::StringDuplication pp(proto); { auto _sapi_statusor61 = (api.pb_duplicate_string(pp.PtrBoth())); if ((__builtin_expect(!_sapi_statusor61.ok(), 0))) { return _sapi_statusor61.status(); } int return_value = std::move(_sapi_statusor61).ValueOrDie();; if (!(return_value)) { return absl::FailedPreconditionError("pb_duplicate_string() failed"); }; } auto _sapi_statusor61 = (pp.GetMessage()); if ((__builtin_expect(!_sapi_statusor61.ok(), 0))) { return _sapi_statusor61.status(); } auto pb_result = std::move(_sapi_statusor61).ValueOrDie();; google::LogMessage( "sandboxed_api/examples/stringop/main_stringop.cc", 61).stream() << "Result PB: " << pb_result.DebugString(); if (!(pb_result.output() == "HelloHello")) { return absl::FailedPreconditionError("Incorrect output"); }; return absl::OkStatus(); }) Expected: is OK Actual: UNAVAILABLE: Could not start the sandbox (of type absl::Status), which is not OK [ FAILED ] StringopTest.ProtobufStringDuplication (19 ms) [ RUN ] StringopTest.ProtobufStringReversal [sandboxed_api/sandbox2/util.cc : 138] RAW: clone(): Operation not permitted [1] [sandboxed_api/sandbox2/forkserver.cc : 559] RAW: Check (pid != -1) failed: failed to fork initial namespaces process: Operation not permitted [1] [sandboxed_api/sandbox2/forkserver.cc : 219] RAW: Receiving init PID from the ForkServer failed E20200710 03:08:35.932246 4262 executor.cc:162] Could not obtain init PID sandboxed_api/examples/stringop/main_stringop.cc:66: Failure Value of: sandbox.Init() Expected: is OK Actual: UNAVAILABLE: Could not start the sandbox (of type absl::Status), which is not OK [ FAILED ] StringopTest.ProtobufStringReversal (7 ms) [ RUN ] StringopTest.RawStringDuplication [sandboxed_api/sandbox2/util.cc : 138] RAW: clone(): Operation not permitted [1] [sandboxed_api/sandbox2/forkserver.cc : 559] RAW: Check (pid != -1) failed: failed to fork initial namespaces process: Operation not permitted [1] [sandboxed_api/sandbox2/forkserver.cc : 219] RAW: Receiving init PID from the ForkServer failed E20200710 03:08:35.938766 4264 executor.cc:162] Could not obtain init PID sandboxed_api/examples/stringop/main_stringop.cc:82: Failure Value of: sandbox.Init() Expected: is OK Actual: UNAVAILABLE: Could not start the sandbox (of type absl::Status), which is not OK [ FAILED ] StringopTest.RawStringDuplication (7 ms) [ RUN ] StringopTest.RawStringReversal [sandboxed_api/sandbox2/util.cc : 138] RAW: clone(): Operation not permitted [1] [sandboxed_api/sandbox2/forkserver.cc : 559] RAW: Check (pid != -1) failed: failed to fork initial namespaces process: Operation not permitted [1] [sandboxed_api/sandbox2/forkserver.cc : 219] RAW: Receiving init PID from the ForkServer failed E20200710 03:08:35.945513 4266 executor.cc:162] Could not obtain init PID sandboxed_api/examples/stringop/main_stringop.cc:98: Failure Value of: sandbox.Init() Expected: is OK Actual: UNAVAILABLE: Could not start the sandbox (of type absl::Status), which is not OK [ FAILED ] StringopTest.RawStringReversal (6 ms) [ RUN ] StringopTest.RawStringLength [sandboxed_api/sandbox2/util.cc : 138] RAW: clone(): Operation not permitted [1] [sandboxed_api/sandbox2/forkserver.cc : 559] RAW: Check (pid != -1) failed: failed to fork initial namespaces process: Operation not permitted [1] [sandboxed_api/sandbox2/forkserver.cc : 219] RAW: Receiving init PID from the ForkServer failed E20200710 03:08:35.952450 4268 executor.cc:162] Could not obtain init PID sandboxed_api/examples/stringop/main_stringop.cc:134: Failure Value of: sandbox.Init() Expected: is OK Actual: UNAVAILABLE: Could not start the sandbox (of type absl::Status), which is not OK [ FAILED ] StringopTest.RawStringLength (7 ms) [ RUN ] StringopTest.RawStringReading [sandboxed_api/sandbox2/util.cc : 138] RAW: clone(): Operation not permitted [1] [sandboxed_api/sandbox2/forkserver.cc : 559] RAW: Check (pid != -1) failed: failed to fork initial namespaces process: Operation not permitted [1] [sandboxed_api/sandbox2/forkserver.cc : 219] RAW: Receiving init PID from the ForkServer failed E20200710 03:08:35.958746 4270 executor.cc:162] Could not obtain init PID sandboxed_api/examples/stringop/main_stringop.cc:144: Failure Value of: sandbox.Init() Expected: is OK Actual: UNAVAILABLE: Could not start the sandbox (of type absl::Status), which is not OK [ FAILED ] StringopTest.RawStringReading (7 ms) [----------] 6 tests from StringopTest (53 ms total)

[----------] Global test environment tear-down [==========] 6 tests from 1 test suite ran. (53 ms total) [ PASSED ] 0 tests. [ FAILED ] 6 tests, listed below: [ FAILED ] StringopTest.ProtobufStringDuplication [ FAILED ] StringopTest.ProtobufStringReversal [ FAILED ] StringopTest.RawStringDuplication [ FAILED ] StringopTest.RawStringReversal [ FAILED ] StringopTest.RawStringLength [ FAILED ] StringopTest.RawStringReading

6 FAILED TESTS`

[cblichmann]

Docker implements its own security policy and uses the default one, when not explicitly told to do otherwise (see the documentation on Seccomp security profiles). So in that sense, this is working as intended.

You will need to both disable Docker's security policy (by using unconfined) and use --privileged in order to run Sandboxed API inside of Docker. The documentation on how to disable is here (same as above).

In short, try this:

docker run --rm -it --security-opt seccomp=unconfined --privileged <DOCKER_IMAGE> <CMD> [ARG...]

[tsosea]

Thanks for the reply! This solution did not work. To reproduce:

sudo docker run --name <name> --rm -it --security-opt seccomp=unconfined --privileged -v <volume> -d ubuntu:18.04 bash -c "sleep 10000000000000000"

Inside Docker: apt update apt install curl gnupg curl https://bazel.build/bazel-release.pub.gpg | apt-key add - echo "deb [arch=amd64] https://storage.googleapis.com/bazel-apt stable jdk1.8" | tee /etc/apt/sources.list.d/bazel.list apt update apt install bazel apt install git openjdk-11-jdk tree python python-pip pip install numpy pip install future apt-get update apt-get install -qy build-essential linux-libc-dev bazel python3 python3-pip libclang-7-dev pip3 install clang git clone https://github.com/google/sandboxed-api && cd sandboxed-api bazel build ... bazel run //sandboxed_api/examples/stringop:main_stringop

The errors stay the same.

[cblichmann]

I tried this out for myself again. Docker also seems to be setting restrictive capabilities on the container. I was able to successfully run the stringop example using this invocation of the container (note the --cap-add=ALL):

docker run -it --rm --security-opt seccomp=unconfined --privileged --cap-add=ALL debian:bullseye

Then I executed the steps above (replacing libclang-7-dev with libclang-dev).

[tsosea]

Unfortunately, this does not work either:

docker run -it --rm --security-opt seccomp=unconfined --privileged --cap-add=ALL debian:bullseye
apt update
apt install curl gnupg
curl https://bazel.build/bazel-release.pub.gpg | apt-key add -
echo "deb [arch=amd64] https://storage.googleapis.com/bazel-apt stable jdk1.8" | tee /etc/apt/sources.list.d/bazel.list
apt update
apt install bazel
apt install git openjdk-11-jdk tree python
apt-get update
apt-get install -qy build-essential linux-libc-dev bazel python3 python3-pip libclang-dev
pip3 install clang
git clone https://github.com/google/sandboxed-api && cd sandboxed-api
bazel build ...
bazel run //sandboxed_api/examples/stringop:main_stringop

[cblichmann]

Same error message? That'd be odd, as I was able to complete those steps successfully.

[tsosea]

Yeah, I am getting the same error message.

[disconnect3d]

Hey,

I'm not familiar with this issue neither I will work on it but I just want to point out some misunderstandings regarding the --privileged docker run flag mentioned in those quotes:

You will need to both disable Docker's security policy (by using unconfined) and use --privileged in order to run Sandboxed API inside of Docker. The documentation on how to disable is here (same as above).

And:

I tried this out for myself again. Docker also seems to be setting restrictive capabilities on the container. I was able to successfully run the stringop example using this invocation of the container (note the --cap-add=ALL):

docker run -it --rm --security-opt seccomp=unconfined --privileged --cap-add=ALL debian:bullseye

Passing --privileged docker run flag does enable all Linux capabilities and disable both seccomp and AppArmor for the contained process[es]. This can be seen in the following runs:

$ docker run --rm -it --privileged ubuntu cat /proc/1/status | egrep -e '(Cap|Sec)'
CapInh:	0000003fffffffff
CapPrm:	0000003fffffffff
CapEff:	0000003fffffffff
CapBnd:	0000003fffffffff
CapAmb:	0000000000000000
Seccomp:	0

$ docker run --rm -it --privileged ubuntu cat /proc/1/attr/current
unconfined

If this wouldn't be true we would not see the following things for pid=1 process:

• CapEff: 0000003fffffffff but instead some other value (3fff.. means all capabilities; note: this can be decoded via capsh --decode=0000003fffffffff tool)
• Seccomp: 0 but instead 2 if seccomp was enabled
• unconfined for the attr/current file, but instead a name of the AppArmor profile and the mode it is run under; this would be docker-default (enforce) by default

As a comparison, lets see those without the --privileged flag:

$ docker run --rm -it ubuntu cat /proc/1/status | egrep -e '(Cap|Sec)'
CapInh:	00000000a80425fb
CapPrm:	00000000a80425fb
CapEff:	00000000a80425fb
CapBnd:	00000000a80425fb
CapAmb:	0000000000000000
Seccomp:	2

$ docker run --rm -it ubuntu cat /proc/1/attr/current
docker-default (enforce)

Additionally, I can say that --privileged also laxes the devices one can access through the devices cgroup v1 controller. This can be seen in the following output:

$ docker run --rm -it ubuntu cat /sys/fs/cgroup/devices/devices.list
b *:* m
c *:* m
c 1:3 rwm
c 1:5 rwm
c 1:7 rwm
c 1:8 rwm
c 1:9 rwm
c 5:0 rwm
c 5:1 rwm
c 5:2 rwm
c 10:200 rwm
c 136:* rwm

$ docker run --rm -it --privileged ubuntu cat /sys/fs/cgroup/devices/devices.list
a *:* rwm

The output format is: <block or char device> <major>:<minor> [rwm] where major:minor are devices major and minor numbers and rwm stands for reading, writing and mknodding a device (yes, the b *:* m and c *:* m in the default Docker configuration means that user can mknod any device file but they will not be able to read or write from it).

I think the only "containment", if we can call it, in --privileged containers are namespaces (which ofc doesn't really protect against anything at this point):

$ sudo ls -la /proc/1/ns
[sudo] password for dc:
total 0
dr-x--x--x 2 root root 0 Apr  7  2021 .
dr-xr-xr-x 9 root root 0 Apr  6  2021 ..
lrwxrwxrwx 1 root root 0 Oct 18 21:40 cgroup -> 'cgroup:[4026531835]'
lrwxrwxrwx 1 root root 0 Oct 18 21:40 ipc -> 'ipc:[4026531839]'
lrwxrwxrwx 1 root root 0 Apr  7  2021 mnt -> 'mnt:[4026531840]'
lrwxrwxrwx 1 root root 0 Oct 18 21:40 net -> 'net:[4026531993]'
lrwxrwxrwx 1 root root 0 Sep 15 06:25 pid -> 'pid:[4026531836]'
lrwxrwxrwx 1 root root 0 Oct 18 21:40 pid_for_children -> 'pid:[4026531836]'
lrwxrwxrwx 1 root root 0 Oct 18 21:40 user -> 'user:[4026531837]'
lrwxrwxrwx 1 root root 0 Oct 18 21:40 uts -> 'uts:[4026531838]'

$ docker run --rm -it --privileged ubuntu ls -la /proc/1/ns
total 0
dr-x--x--x 2 root root 0 Oct 18 20:40 .
dr-xr-xr-x 9 root root 0 Oct 18 20:40 ..
lrwxrwxrwx 1 root root 0 Oct 18 20:40 cgroup -> 'cgroup:[4026531835]'
lrwxrwxrwx 1 root root 0 Oct 18 20:40 ipc -> 'ipc:[4026532357]'
lrwxrwxrwx 1 root root 0 Oct 18 20:40 mnt -> 'mnt:[4026532355]'
lrwxrwxrwx 1 root root 0 Oct 18 20:40 net -> 'net:[4026532360]'
lrwxrwxrwx 1 root root 0 Oct 18 20:40 pid -> 'pid:[4026532358]'
lrwxrwxrwx 1 root root 0 Oct 18 20:40 pid_for_children -> 'pid:[4026532358]'
lrwxrwxrwx 1 root root 0 Oct 18 20:40 user -> 'user:[4026531837]'
lrwxrwxrwx 1 root root 0 Oct 18 20:40 uts -> 'uts:[4026532356]'

$ docker run --rm -it ubuntu ls -la /proc/1/ns
total 0
dr-x--x--x 2 root root 0 Oct 18 20:41 .
dr-xr-xr-x 9 root root 0 Oct 18 20:41 ..
lrwxrwxrwx 1 root root 0 Oct 18 20:41 cgroup -> 'cgroup:[4026531835]'
lrwxrwxrwx 1 root root 0 Oct 18 20:41 ipc -> 'ipc:[4026532357]'
lrwxrwxrwx 1 root root 0 Oct 18 20:41 mnt -> 'mnt:[4026532355]'
lrwxrwxrwx 1 root root 0 Oct 18 20:41 net -> 'net:[4026532360]'
lrwxrwxrwx 1 root root 0 Oct 18 20:41 pid -> 'pid:[4026532358]'
lrwxrwxrwx 1 root root 0 Oct 18 20:41 pid_for_children -> 'pid:[4026532358]'
lrwxrwxrwx 1 root root 0 Oct 18 20:41 user -> 'user:[4026531837]'
lrwxrwxrwx 1 root root 0 Oct 18 20:41 uts -> 'uts:[4026532356]'

Also, if you pass in --privileged you cannot e.g. drop capabilities; I am not sure about Seccomp or AppArmor though. FWIW I have created a GH issue about a lack of warning on this matter in the Docker project in https://github.com/moby/moby/issues/42751.

As a bonus, I have also elaborated about some of those in this presentation.