for-mac
for-mac copied to clipboard
docker
Docker scout huge memory consumption
Description
After updating to the latest docker, I noticed that my system (MacBook Pro M1, 16 GB) runs out of working memory when building docker images. Checking Activity Monitor reveals a huge memory consumption by the process docker-scout:
This process starts shortly after the image is built, so it seems like docker-scout tries to analyse the built images. However I do not think I ever enabled docker-scout. Actually, I was not even aware about its existence. The docker scout configuration is empty, however there is clearly something in the cache:
% docker scout config
Configuration empty. See docker scout config --help for details.
% docker scout cache df
...
Total: 5.0 GB
Other commands like docker scout quickview ask me for my Docker ID, which I do not have.
Please, make it possible to disable docker-scout completely!
Reproduce
- Create a Dockerfile:
FROM jupyter/minimal-notebook:2023-09-18 AS test
RUN touch /tmp/test
- Run
docker build -t test .from the same directory - Observe the memory consumption spikes after the image has been built:
Expected behavior
I expect that Docker should not start any new processes (or increase the memory consumption) after the image is built but not used.
docker version
Client:
Cloud integration: v1.0.35+desktop.4
Version: 24.0.6
API version: 1.43
Go version: go1.20.7
Git commit: ed223bc
Built: Mon Sep 4 12:28:49 2023
OS/Arch: darwin/arm64
Context: desktop-linux
Server: Docker Desktop 4.23.0 (120376)
Engine:
Version: 24.0.6
API version: 1.43 (minimum version 1.12)
Go version: go1.20.7
Git commit: 1a79695
Built: Mon Sep 4 12:31:36 2023
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.6.22
GitCommit: 8165feabfdfe38c65b599c4993d227328c231fca
runc:
Version: 1.1.8
GitCommit: v1.1.8-0-g82f18fe
docker-init:
Version: 0.19.0
GitCommit: de40ad0
docker info
Client:
Version: 24.0.6
Context: desktop-linux
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.11.2-desktop.4
Path: /Users/username/.docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.21.0-desktop.1
Path: /Users/username/.docker/cli-plugins/docker-compose
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.0
Path: /Users/username/.docker/cli-plugins/docker-dev
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.20
Path: /Users/username/.docker/cli-plugins/docker-extension
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v0.1.0-beta.7
Path: /Users/username/.docker/cli-plugins/docker-init
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: /Users/username/.docker/cli-plugins/docker-sbom
scan: Docker Scan (Docker Inc.)
Version: v0.26.0
Path: /Users/username/.docker/cli-plugins/docker-scan
scout: Command line tool for Docker Scout (Docker Inc.)
Version: 0.24.1
Path: /Users/username/.docker/cli-plugins/docker-scout
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 3
Server Version: 24.0.6
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
runc version: v1.1.8-0-g82f18fe
init version: de40ad0
Security Options:
seccomp
Profile: unconfined
cgroupns
Kernel Version: 6.3.13-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 5.8GiB
Name: docker-desktop
ID: eb28bdcf-66a4-4963-bbcf-c58cf711d1a9
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5555
127.0.0.0/8
Live Restore Enabled: false
WARNING: daemon is not using the default seccomp profile
Diagnostics ID
6734B03B-C87B-4461-884C-139350A00371/20230921080327
Additional Info
Interestingly, I did not notice docker-scout starting if I use other base docker images such as ubuntu:latest, debian:latest or alpine:latest. Is this behaviour image/organization-specific?
The problem is exacerbated if several images are built at once, e.g., using docker compose.
For example for this Dockerfile:
FROM jupyter/minimal-notebook:2023-09-18 AS test1
RUN touch /tmp/test
FROM test1 AS test2
RUN touch /tmp/test
FROM test2 AS test3
RUN touch /tmp/test
and this docker-compose.yaml:
version: '3'
services:
test1:
image: test1
build:
target: test1
test2:
image: test2
build:
target: test2
test3:
image: test3
build:
target: test3
after running docker compose build, two docker-scout processes start and memory consumption almost doubles:
I had a similar situation in my first screenshot.
Furthermore, when all system memory is consumed, docker crashes, leaving some temporary files in /private/var/folders/, which, in my case occupied 20+ GB. First I did not notice this and my system ran out of disk space.
You can disable the SBOM background indexer under Settings > Features in development > Experimental features > Enable background SBOM indexing
Thanks @bsousaa, that worked! Finally the memory is under control! Did this feature appear in the last version? Please consider disabling SBOM by default.
Just had this disable a macOS machine. Why are experimental features enabled by default?
This absolutely destroyed my 8GB MBA (I know... my fault!) but would be good to have this disabled by default!
@MattJustMatt Same - knew this machine had little memory but have rarely seen it slow to the absolute crawl this feature caused!
Just to add that it also occurs in Docker Desktop for Windows, the experimental feature is activated by default and the same workaround as proposed here fixes it.
I've just installed the latest version of Docker Desktop 4.25.0 and this feature SBOM indexing has been enabled by default😒
Docker team, please disable it by default🙏
On my MacBook m1 building time increased 10x times! I've waited to make a build for around 35 minutes instead of 3 after I disable it.
This is no longer found under "experimental features" but is still called "SBOM indexing" under the general settings tab. Until I disabled it, 2 docker-scout processes were consuming all 24GB of memory on my M2 macbook, and I had to shut it off to get any work done.
I agree that this should be disabled by default.
I just updated to docker desktop 4.25.2 (129061) to observe that SBOM indexing has been reenabled after I manually disabled it in the settings:
What is going on??
➕ on this issue. Had the same problem when building a simple postgres image. Memory consumption going to its limits with the container unused. Lots of memory usage also happening with docker-scout process.
This was using 15GB of RAM out of a 16GB Intel Mac on 4.25.0. This needs attention.
For anyone still finding this issue, the setting in Docker Desktop has now moved to Settings > General > SBOM Indexing (tested on 4.25.4).
Chiming in here:
I have disabled SBOM indexing, and Usage Statistics and restarted docker desktop. docker-scout is still being run aggressively, as if on a timer even if no new containers are pulled or built, and it is trying to use up to 32 GB (!!!) of memory in one of two processes on a computer with only 16.
The only way I can somewhat attempt to use my work computer for work right now is running killall -9 docker-scout repeatedly on the command line to buy myself 15 minutes of actually using my computer.
This is unacceptable.
Didn't realize this was happening till my machine was running slow. Docker Scout is throttling entire machine performance. Is there a timeline on this fix since it seems like a High priority bug since it affects entire machines.
@OsirisMaeluma memory consumption will depend on the size of the images in your machine. You can disable indexing in Settings > General > SBOM Indexing
What's the point of a memory usage limit if Docker is simply going to blow past it, with a 4GB memory limit, and Docker's scout component alone using 6GB.
Chiming in here:
I have disabled SBOM indexing, and Usage Statistics and restarted docker desktop.
docker-scoutis still being run aggressively, as if on a timer even if no new containers are pulled or built, and it is trying to use up to 32 GB (!!!) of memory in one of two processes on a computer with only 16.The only way I can somewhat attempt to use my work computer for work right now is running
killall -9 docker-scoutrepeatedly on the command line to buy myself 15 minutes of actually using my computer.This is unacceptable.
So, this is interesting. After a few system restarts, it seems like Docker desktop is now honoring those settings and not firing up docker-scout. Restarting Docker Desktop isn't enough though, despite the UI implying it is. Apparently these settings in particular only take on a full reboot.
FWIW, it stopped immediately after unchecking the box for me
On Tue, 6 Feb 2024 at 7:26 pm, Jason DePeaux @.***> wrote:
Chiming in here:
I have disabled SBOM indexing, and Usage Statistics and restarted docker desktop. docker-scout is still being run aggressively, as if on a timer even if no new containers are pulled or built, and it is trying to use up to 32 GB (!!!) of memory in one of two processes on a computer with only 16.
The only way I can somewhat attempt to use my work computer for work right now is running killall -9 docker-scout repeatedly on the command line to buy myself 15 minutes of actually using my computer.
This is unacceptable.
So, this is interesting. After a few system restarts, it seems like Docker desktop is now honoring those settings and not firing up docker-scout. Restarting Docker Desktop isn't enough though, despite the UI implying it is. Apparently these settings in particular only take on a full reboot.
— Reply to this email directly, view it on GitHub https://github.com/docker/for-mac/issues/6987#issuecomment-1930611091, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC5QOWC4JQGVMTDASF4SRATYSJ7WFAVCNFSM6AAAAAA5BDYNXSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZQGYYTCMBZGE . You are receiving this because you commented.Message ID: @.***>
Oh god, I've just discovered that thing. It's bloody awful. I have quite big build image (around 50gb of code) and decided to build it on Docker Desktop (usually do it in Engine on some Linux distribution like fedora). Docker Engine on Linux used 9 gigs entirely (with the system!). And build on Desktop with scout ballooned to... 110gigs. Of ram. When pulling an image. This is atrocious and it should be disabled by default.
This is still a massive problem. Computers just grind to a halt because of 97% memory usage because of docker scout. That, and the fact that disabling it often does not work at all makes Docker Desktop at this moment pretty unusable unless you have task manager open to terminate docker scout every few minutes.
Devs, please look at this and at least DISABLE it by default since it is causing so much problems.
It is wild that this is still enabled by default. I have a 32 GB machine and it almost froze up completely yesterday, until I discovered that docker-scout.exe was taking up multiple GBs of RAM and I found this issue here. How could an experimental feature like this even be promoted to a stable feature, when it is clearly broken?
Just managed to catch it after last time it wasn't visible in the task manager. This started eating over 16gb of ram on a 32gb machine, nearly crashing it. This issue also has been open for 8 months. At this point, this is just malicious.
No joke, it just used up all of my 64GB(!!!!!) of RAM on my windows machine. And I am running the normal Nvidia PyTorch containers (nvcr.io/nvidia/pytorch). SBOM Indexing in its current form is a hazardous feature! Fix it, disable it, or get rid of it!
