for-linux docker-ce 19.03: domainname denied if userns enabled

[x] This is a bug report
[ ] This is a feature request
[ ] I searched existing issues before opening this one

Expected behavior

When docker is user-namespace enabled, starting a container withdocker run -ti --domainname my.domain debian /bin/bash should work.

Actual behavior

docker run fails with:

# docker run -ti --domainname my.domain debian /bin/bash
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"write sysctl key kernel.domainname: open /proc/sys/kernel/domainname: permission denied\"": unknown.

Info: it worked on docker-ce version 18.09 and 18.03

Steps to reproduce the behavior

Run docker-daemon with 'userns-remap' = 'default'

# docker run -ti --domainname my.domain debian /bin/bash

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.1
 API version:       1.40
 Go version:        go1.12.5
 Git commit:        74b1e89
 Built:             Thu Jul 25 21:22:03 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.1
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.5
  Git commit:       74b1e89
  Built:            Thu Jul 25 21:20:35 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 16
  Running: 13
  Paused: 0
  Stopped: 3
 Images: 34
 Server Version: 19.03.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: journald
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
  userns
 Kernel Version: 4.9.0-9-amd64
 Operating System: Debian GNU/Linux 9 (stretch)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 62.71GiB
 Name: host2
 ID: VO7E:SES6:T7W5:FZNZ:QVD5:EQQJ:6BIS:BH4A:NQZM:3UHF:DYYL:7Z32
 Docker Root Dir: /var/lib/docker/100000.100000
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.) docker-ce is running on a physical server. It is running with the following daemon-options:

{
	"userns-remap": "default",
	"userland-proxy": false,
	"log-driver": "journald",
	"log-opts": {
		"tag": "[{{.ImageName}}]{{.Name}}"
	},
	"live-restore": false,
	"dns-opts": ["timeout:3"],
	"default-ipc-mode": "private"
}

subuid/subgid:

# cat /etc/subuid
dockremap:100000:65536
# cat /etc/subgid
dockremap:100000:65536

Jul 29 '19 11:07 lysandef

Thanks for reporting. I believe this is caused by https://github.com/moby/moby/pull/37302

@cyphar @crosbymichael

This seems to be the case because sysctls get applied after the userns is setup?

Jul 29 '19 17:07 cpuguy83

Yes, this looks like a runc bug -- however I'm fairly sure the issue is not that it's done after the userns is setup but instead that it's done before it's setup. To write to kernel.domainname you need to have privileges in the associated UTS namespace and so it looks to me like we are setting sysctls too early instead of too late.

I'll take a look at this. opencontainers/runc#2091 is the tracking issue I opened for it.

Jul 30 '19 04:07 cyphar

Just ran into this with the gitlab image after upgrading docker to the version OP has.

Are there no other workaround other than either disabling namespaces or downgrading docker?

Aug 24 '19 15:08 zingmars

I don't think an image can have the domain encoded in it. So you could also just not art domain name on your container.

On Sat, Aug 24, 2019 at 08:11 Ingmars Melkis [email protected] wrote:

Just ran into this with the gitlab image after upgrading docker to the version OP has.

Are there no other workaround other than either disabling namespaces or downgrading docker?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/docker/for-linux/issues/743?email_source=notifications&email_token=AAGDCZQJGQ53CP2NWHFQVHTQGFFTDA5CNFSM4IHRJXRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5CB7XI#issuecomment-524558301, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGDCZVUCANKWAX5XRSTOTTQGFFTDANCNFSM4IHRJXRA .

--

Brian Goff

Aug 24 '19 16:08 cpuguy83

I'm using a simple docker-compose and I never even knew that setting a domain name is possible. I don't know what's setting it, but I'm not setting it manually.

Aug 24 '19 16:08 zingmars

Can you post an example of your compose yaml?

On Sat, Aug 24, 2019 at 09:54 Ingmars Melkis [email protected] wrote:

I'm using a simple docker-compose and I never even knew that setting a domain name is possible. I don't know what's setting it, but I'm not setting it manually.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/docker/for-linux/issues/743?email_source=notifications&email_token=AAGDCZRMEAV5GH263DDMNETQGFRTRA5CNFSM4IHRJXRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5CDXQQ#issuecomment-524565442, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGDCZXTCRNHMSFMTP2OUVDQGFRTRANCNFSM4IHRJXRA .

--

Brian Goff

Aug 24 '19 16:08 cpuguy83

version: '2.3'
services:
  web:
  mem_limit: 4g
  image: 'gitlab/gitlab-ce:latest'
  restart: always
  hostname: '<hostname>'
  environment:
    GITLAB_OMNIBUS_CONFIG: | (...)
  ports:
    -  '127.0.0.1:8000:8181'
    -  '127.0.0.1:2222:22'
  volumes:
    - '/var/log/gitlab/:/var/log/gitlab'
    - '/var/www/gitlab/data/:/var/opt/gitlab'

The only thing that I can think of is hostname doing the same thing as domainname. If anything, domainname is so poorly documented that I still have no idea what exactly it sets.

Aug 24 '19 17:08 zingmars

It's set via the "hostname" parameter.

On Sat, Aug 24, 2019 at 10:04 Ingmars Melkis [email protected] wrote:

version: '2.3' services: web: mem_limit: 4g image: 'gitlab/gitlab-ce:latest' restart: always hostname: '' environment: GITLAB_OMNIBUS_CONFIG: | (...) ports: - '127.0.0.1:8000:8181' - '127.0.0.1:2222:22' volumes: - '/var/log/gitlab/:/var/log/gitlab' - '/var/www/gitlab/data/:/var/opt/gitlab'

The only thing that I can think of is hostname doing the same thing as domainname. If anything, domainname is so poorly documented that I still have no idea what exactly it sets.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/docker/for-linux/issues/743?email_source=notifications&email_token=AAGDCZVPZQZAGNURKTU2F73QGFS2XA5CNFSM4IHRJXRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5CD4WI#issuecomment-524566105, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGDCZS4ALF5AUHHFBUJKI3QGFS2XANCNFSM4IHRJXRA .

--

Brian Goff

Aug 24 '19 17:08 cpuguy83

Hello from 2020 here. This is still the problem with Server Version: 19.03.13 and userns enabled....

'Error 400: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"write sysctl key kernel.domainname: open /proc/sys/kernel/domainname: permission denied\"": unknown'

Oct 02 '20 01:10 b-a-t

Same is apparently true for Docker 20.10 I'm using docker in docker using Sysbox which has userns enabled.

400 Client Error for http+docker://localhost/v1.41/containers/3dfc85316a4c09b481f56bf1f2f22e6491083969615c859ad0b11407d7bb508d/start: Bad Request ("OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: write sysctl key kernel.domainname: open /proc/sys/kernel/domainname: permission denied: unknown")

admin@syscont:~$ docker version
Client: Docker Engine - Community
 Version:           20.10.1
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        831ebea
 Built:             Tue Dec 15 04:34:58 2020
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.1
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       f001486
  Built:            Tue Dec 15 04:32:52 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Dec 17 '20 21:12 jo-me

I observed this problem on proxmox/unprivileged LXC container with Docker (in my case, Gitlab via docker-compose)

pve-manager/6.4-8
kernel: 5.4.114-1-pve
Docker version 20.10.7

Solved by simply removing/commenting hostname from the docker-compose.yml:

web:
  image: 'gitlab/gitlab-ce:latest'
  restart: always
  # hostname: 'gl.local.mytld.com'
  environment:
    GITLAB_OMNIBUS_CONFIG: |
      external_url 'https://gl.local.mytld.com'
      ...
      registry_external_url 'https://registry.local.mytld.com'
      ...
  ports:
  ...

It works flawlessly without hostname - I assume since there is also external_url.

Note that I have a reverse proxy (nginx) configured.