buildkit icon indicating copy to clipboard operation
buildkit copied to clipboard
verified publisher icon moby

buildkit/buildx: --cpuset-cpus ignored

Open thomas-riccardi opened this issue 6 years ago • 5 comments

Description

I am trying to limit CPUs exposed during build (to limit build parallelism in make -j$(nproc)). docker build --cpuset-cpus=0-3 does work, but only with the old builder. With buildkit or buildx it seems to be ignored

Steps to reproduce the issue:

  1. Dockerfile: 
    FROM ubuntu:18.04
RUN nproc
  2. docker build --cpuset-cpus=0-3 . --no-cache --progress=plain

Describe the results you received:

  • old builder: 4: OK
$ DOCKER_BUILDKIT=0 docker build --cpuset-cpus=0-3 . --no-cache --progress=plain
Sending build context to Docker daemon  3.072kB
Step 1/2 : FROM ubuntu:18.04
 ---> 4c108a37151f
Step 2/2 : RUN nproc
 ---> Running in be67670148e7
4
Removing intermediate container be67670148e7
 ---> ea93757367b3
Successfully built ea93757367b3
  • buildkit: 16: KO
4 DOCKER_BUILDKIT=1 docker build --cpuset-cpus=0-3 . --no-cache --progress=plain
moby/moby#2 [internal] load build definition from Dockerfile
moby/moby#2 transferring dockerfile: 71B done
moby/moby#2 DONE 0.0s

moby/moby#1 [internal] load .dockerignore
moby/moby#1 transferring context: 2B done
moby/moby#1 DONE 0.0s

moby/moby#3 [internal] load metadata for docker.io/library/ubuntu:18.04
moby/moby#3 DONE 0.0s

moby/moby#4 [1/2] FROM docker.io/library/ubuntu:18.04
moby/moby#4 CACHED

moby/moby#5 [2/2] RUN nproc
moby/moby#5 0.517 16
moby/moby#5 DONE 0.6s

moby/moby#6 exporting to image
moby/moby#6 exporting layers done
moby/moby#6 writing image sha256:7c1e0680e16550a3d0f098d94ca98d2afeedc45ff52235c4a5c8e1d02700276c done
moby/moby#6 DONE 0.0s
  • buildx: 16: KO
$ docker buildx build --cpuset-cpus=0-3 . --no-cache --progress=plain
moby/moby#2 [internal] load .dockerignore
moby/moby#2 transferring context: 2B done
moby/moby#2 DONE 0.0s

moby/moby#1 [internal] load build definition from Dockerfile
moby/moby#1 transferring dockerfile: 65B done
moby/moby#1 DONE 0.0s

moby/moby#3 [internal] load metadata for docker.io/library/ubuntu:18.04
moby/moby#3 DONE 0.0s

moby/moby#4 [1/2] FROM docker.io/library/ubuntu:18.04
moby/moby#4 CACHED

moby/moby#5 [2/2] RUN nproc
moby/moby#5 0.550 16
moby/moby#5 DONE 0.7s

moby/moby#6 exporting to image
moby/moby#6 exporting layers 0.0s done
moby/moby#6 writing image sha256:ba2c976d12f5a9f352144a02f7cb604214816de7f138cbe4fa61267cc0333fce done
moby/moby#6 DONE 0.0s
  • buildx with non-default builder: 16: KO
$ docker buildx create  --use
blissful_williams
$ docker buildx build --cpuset-cpus=0-3 . --no-cache --progress=plain
$ docker buildx build --cpuset-cpus=0-3 . --no-cache --progress=plain
WARN[0000] No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load 
moby/moby#1 [internal] booting buildkit
moby/moby#1 pulling image moby/buildkit:buildx-stable-1
moby/moby#1 pulling image moby/buildkit:buildx-stable-1 1.5s done
moby/moby#1 creating container buildx_buildkit_blissful_williams0
moby/moby#1 creating container buildx_buildkit_blissful_williams0 0.5s done
moby/moby#1 DONE 2.0s

moby/moby#2 [internal] load build definition from Dockerfile
moby/moby#2 transferring dockerfile:
moby/moby#2 transferring dockerfile: 65B done
moby/moby#2 DONE 0.1s

moby/moby#3 [internal] load .dockerignore
moby/moby#3 transferring context: 2B done
moby/moby#3 DONE 0.1s

moby/moby#4 [internal] load metadata for docker.io/library/ubuntu:18.04
moby/moby#4 DONE 1.5s

moby/moby#5 [1/2] FROM docker.io/library/ubuntu:18.04@sha256:6e9f67fa63b0323e9a1e587...
moby/moby#5 resolve docker.io/library/ubuntu:18.04@sha256:6e9f67fa63b0323e9a1e587fd71c561ba48a034504fb804fd26fd8800039835d done
moby/moby#5 sha256:6e9f67fa63b0323e9a1e587fd71c561ba48a034504fb804fd26fd8800039835d 1.42kB / 1.42kB done
moby/moby#5 sha256:134c7fe821b9d359490cd009ce7ca322453f4f2d018623f849e580a89a685e5d 1.15kB / 1.15kB done
moby/moby#5 sha256:45d437916d5781043432f2d72608049dcf74ddbd27daa01a25fa63c8f1b9adc4 162B / 162B 0.1s done
moby/moby#5 sha256:7ddbc47eeb70dc7f08e410a6667948b87ff3883024eb41478b44ef9a81bf400c 0B / 26.69MB 0.1s
moby/moby#5 sha256:c1bbdc448b7263673926b8fe2e88491e5083a8b4b06ddfabf311f2fc5f27e2ff 0B / 35.36kB 0.1s
moby/moby#5 sha256:775349758637aff77bf85e2ff0597e86e3e859183ef0baba8b3e8fc8d3cba51c 3.41kB / 3.41kB done
moby/moby#5 sha256:8c3b70e3904492c753652606df4726430426f42ea56e06ea924d6fea7ae162a1 0B / 845B 0.1s
moby/moby#5 sha256:c1bbdc448b7263673926b8fe2e88491e5083a8b4b06ddfabf311f2fc5f27e2ff 35.36kB / 35.36kB 0.2s done
moby/moby#5 sha256:7ddbc47eeb70dc7f08e410a6667948b87ff3883024eb41478b44ef9a81bf400c 5.48MB / 26.69MB 0.4s
moby/moby#5 sha256:7ddbc47eeb70dc7f08e410a6667948b87ff3883024eb41478b44ef9a81bf400c 15.42MB / 26.69MB 0.6s
moby/moby#5 sha256:8c3b70e3904492c753652606df4726430426f42ea56e06ea924d6fea7ae162a1 845B / 845B 0.4s done
moby/moby#5 sha256:7ddbc47eeb70dc7f08e410a6667948b87ff3883024eb41478b44ef9a81bf400c 26.69MB / 26.69MB 0.8s done
moby/moby#5 unpacking docker.io/library/ubuntu:18.04@sha256:6e9f67fa63b0323e9a1e587fd71c561ba48a034504fb804fd26fd8800039835d
moby/moby#5 unpacking docker.io/library/ubuntu:18.04@sha256:6e9f67fa63b0323e9a1e587fd71c561ba48a034504fb804fd26fd8800039835d 1.1s done
moby/moby#5 DONE 2.0s

moby/moby#6 [2/2] RUN nproc
moby/moby#6 0.180 16
moby/moby#6 DONE 0.3s

Describe the results you expected: I would expect to get nproc=4 on at least DOCKER_BUILDKIT=1 docker build --cpuset-cpus=0-3, and ideally on buildx builds too.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea838
 Built:             Wed Nov 13 07:29:52 2019
 OS/Arch:           linux/amd64
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea838
  Built:            Wed Nov 13 07:28:22 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

Client:
 Debug Mode: false
 Plugins:
  app: Docker Application (Docker Inc., v0.8.0)
  buildx: Build with BuildKit (Docker Inc., v0.3.1-tp-docker)

Server:
 Containers: 134
  Running: 7
  Paused: 0
  Stopped: 127
 Images: 143
 Server Version: 19.03.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.15.0-70-generic
 Operating System: Ubuntu 18.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 31.43GiB
 Name: thomas-MS-7B86
 ID: YJPE:ZYR6:6MPI:CUXV:5UJI:4PO7:OYIH:BG3U:574D:BOAM:JSQ3:J2FD
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: thomasriccardi
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.): Desktop with AMD Ryzen 7 2700X Eight-Core Processor

thomas-riccardi avatar Nov 27 '19 11:11 thomas-riccardi

In buildx these flags were added as hidden by https://github.com/docker/buildx/commit/103b452876cb7bbcec83aed47b2ea28bcae321ec and thus don't raise any error, but they don't seem to be used later on: silent error...

thomas-riccardi avatar Nov 27 '19 11:11 thomas-riccardi

transferred from moby/moby#40260

AkihiroSuda avatar Nov 27 '19 12:11 AkihiroSuda

is this related to https://github.com/moby/buildkit/issues/1131 ?

FernandoMiguel avatar Nov 27 '19 12:11 FernandoMiguel

No, #1131 is for limiting concurrency of LLB solver routines.

cgroup (https://github.com/moby/buildkit/issues/593) is exactly related.

AkihiroSuda avatar Nov 27 '19 12:11 AkihiroSuda

I've found that using a non-default builder that specifies the cpuset on creation will work

FROM alpine
RUN nproc

docker buildx create --driver-opt=cpuset-cpus=0-1 --name=nproc-test
docker buildx build --builder=nproc-test . --progress=plain --no-cache
docker buildx rm nproc-test

Will output

#5 [2/2] RUN nproc
#5 0.037 2
#5 DONE 0.0s

EricChen1248 avatar Nov 10 '25 15:11 EricChen1248