[Docker Desktop] Desktop resource clean up

[nebuk89]

Tell us about your request Provide a UI tool to show users how much of their disk space the different assets in Desktop are taking up (images, containers, cache) and a tool to remove these to free up space

Which service(s) is this request for? Desktop

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? We hear a lot of feedback that 'desktop is using all of my disk space' and we would like to help our users understand why and address this

Are you currently working around this issue? N/A Additional context N/A Attachments

[mjambon]

I also have frequently full disks on my development machines as well as our CI hosts running Jenkins, where the solution is to clear the cache nightly with a cron job, requiring base images to be downloaded again and again, defeating the point of having a cache.

I asked on ServerFault if there was a way to delete unused objects but keep the ones that have been recently used (I haven't received an answer at this time). I think it would make the cache more effective and greatly enhance the user experience when disk space is limited (it usually is) and network is slow.

I don't know anything about Docker internals but I imagine that tracking the usage of image layers would be a first step toward detecting unused objects worth deleting. Usage could be based on container lifetime, i.e. the termination timestamp of the last container that used a certain image layer. An approximate date would already be useful for caching purposes.

Food for thought.

Edit: I now realize that Docker Desktop is a specific part of the product, which is not particularly tied to the comments I made. Please let me know if should create a separate issue or more generally how to be helpful.

[ingshtrom]

@mjambon I agree with you. In the meantime, have you looked at using docker system prune with a label filter? In our infra we've used two prune jobs in the past, one for everything, and then a specific one that allowed keeping specially tagged items longer.

/usr/bin/docker system prune -a -f --filter "label!=com.docker.ucp.version" --filter "until=120m" would delete anything that does not have the com.docker.ucp.version label AND is older than 120 minutes.

/usr/bin/docker container prune -f --filter "until=120m" would prune all stopped containers older than 120 minutes. Thus, we would be left with running containers, and all images/networks/dangling images/dangling build cache items that have been used within 120 minutes and have the com.docker.ucp.version label.

So you could tag your images/containers/etc with dev=true and run /usr/bin/docker system prune -a -f --filter "label=dev" --filter "until=30m" and /usr/bin/docker system prune -a -f --filter "label!=dev" --filter "until=120m". This would get rid of dev items taking up storage space much quicker, while keeping around other stuff.

[tiborvass]

I don't think this is a desktop issue. All installations have this problem.

[stephen-turner]

@tiborvass You're right, all installations potentially have this problem, but AIUI this issue is specifically about exposing the existing docker prune via Desktop UI. Enhancements to docker prune maybe belong in a separate ticket.

[thaJeztah]

For Docker Desktop, it would be great to also have an overview how disk space is consumed. There's parts that cannot be removed inside the container (e.g., docker and containerd itself), and parts that can be cleaned up (e.g. through docker system prune).

Having an indicator to see what takes up space would be useful; something similar to macOS's presentation;

For cleaning up; I think both "automatic" and "manual" could make sense. For automatic cleaning, some care should be taken, because it's harder to decide if something is "ok" to be removed. When using BuildKit as a builder, build-cache can be automatically "garbage collected", and provides some configuration to customise (see https://github.com/moby/moby/pull/37846)

For other resources both "manual" and "automatic" should default to NOT remove volumes, as they may contain important data (even if unused by a container, we cannot assume they can be cleaned up).

[ Cleanup resources ]

WARNING! This will remove:
  - all stopped containers
  - all networks not used by at least one container
  - all dangling images
  - all dangling build cache

Possibly have checkboxes to customize. For example, docker image prune (by default) only cleans up "untagged images", but allows using --all to remove all images that are not currently used by a container, and docker builder prune allows "keeping" X gigabytes of cache and/or only removing cache that hasn't been used longer in the last X hours

Cleanup resources

[ ] customize
  [x] dangling images
    [ ] all unused images
  [x] stopped containers
  [x] build-cache
  [x] unused networks
  [ ] unused volumes

[tiborvass]

Yes but the feature first needs to be in engine anyway via daemon settings which would out-of-the-box work in Desktop via the daemon settings pane. Happy to have nicer UI on top of that, but I would argue that's not the highest priority. Priority is to have the feature in the first place.

[JimCodified]

Does Docker Desktop do any automatic clean up today? If so, part of the UI that would be nice is to show objects queued for cleanup in the near future. Maybe also a log of what has been cleaned up, if there is any automation.

[stephen-turner]

@JimCodified Desktop does not do any automated clean-up, and I'm not sure we'd want to do that either because we'd be deleting the user's files. Certainly in the first version it would just be about reporting space better and allowing the user to select to clean it up.

[yurii-sio2]

The problem still exists. Scripts for pruning didn't help. Win10, LSTC 2018, All updates installed.

[carehart]

I realize this could be regarded as a separate issue, but it's definitely related to "resource cleanup": it sure would be nice if DD had a ui feature to move itself (or at least the vm location) from one drive to another.

People often install it and only find over time after many pulls and builds that DD is filling their drive--often their system drive, to great peril. And it's not trivial to move.

Yes, there's the experimental ui and its keywords (if you know them, and accurately track their changes over the years). But a single button click--which knew how to handle things for us (on per-os basis, or for Wsl vs WSL2, etc) would sure make life easier for many.

[thaJeztah]

@carehart is the "Disk image location" option in the preferences what you're looking for? https://docs.docker.com/docker-for-windows/#resources

Disk image location: Specify the location of the Linux volume where containers and images are stored.

You can also move the disk image to a different location. If you attempt to move a disk image to a location that already has one, you get a prompt asking if you want to use the existing image or replace it.

[thaJeztah]

Linking another ticket that's somewhat related (switching default logging driver to reduce disk space taken up by container logs); https://github.com/docker/roadmap/issues/163

[carehart]

@carehart is the "Disk image location" option in the preferences what you're looking for? https://docs.docker.com/docker-for-windows/#resources

Disk image location: Specify the location of the Linux volume where containers and images are stored. You can also move the disk image to a different location. If you attempt to move a disk image to a location that already has one, you get a prompt asking if you want to use the existing image or replace it.

That works for those on Hyper-V, yes (and I see I missed it was added/moved from being an experimental arg instead). But for those using WSL/WSL2, it doesn't appear. And sure, someone may say "take up the problem with MS", I'll just say that if DD creates the image, then it's reasonable to ask it (as part of this "cleanup" feature suggestion) that it (DD) could help with that use case as well.

But I will understand if the decision is made that it feels not DD's responsibility. Just wanted to put it on the table for consideration.

[stephanierifai]

Hi all! Somewhat relatedly to this item we have been prototyping two things we might build:

1. Improving the clean up interface that we have on images and also making it available for the volume management interface

2. Multi select check boxes on images and volumes for easy removal without having to enter the clean up mode (note that we plan on adding the disk space saved on this screen too like in clean up screen above).

With the goal of allowing you more control for freeing up space that is being used by volumes. Some mock ups attached and we would love any feedback you have (What do you like/not like? What’s missing from making this useful? Is anything confusing?)

Curious if there is any interest in having a clean up mode across resources rather than doing it separately?

[stephanierifai]

Also, based on the other feedback on this ticket, it sounds like there is a need for:

• Overall more visibility into what resources take up space
• The ability to set policies for automatic clean up on some frequency
• Exposing more of the prune capabilities within the Docker Dashboard (the UI) for example delete images older than X minutes

Are any the most important to you all on this? Are there other problems you want solved related to clean up not yet mentioned?

[carehart]

... Are there other problems you want solved related to clean up not yet mentioned?

Thanks for asking. I'll ping with a reminder about the point I'd made above about helping folks (wanting to clean up DD-related disk usage) with an easy way to move all the images to another drive, especially when running DD for Win with wsl 2. Or might anyone say that's been resolved since my last comment here, a year ago: https://github.com/docker/roadmap/issues/13#issuecomment-742829877?

[stephanierifai]

Hi all! In 4.8 we shipped a disk usage extension to help with visibility and reclaiming space. We'd love to hear your feedback!

[thaJeztah]

Love the new feature 😃

The round / circular diagram is quite "hungry" on screen real-estate; I wonder if it has been considered to use a horizontal diagram instead, as I posted in my older comment above (matching macOS's approach); https://github.com/docker/roadmap/issues/13#issuecomment-631382934

Doing so would likely not allow to have the "legend", which could be solved using a hover box;