Investigate on docker overlay volume size (+Solution)

[ignazio-bovo]

I have observed that on my server the /var/lib/docker used as mount point for the overlay volume is occupying 850GB that is all the available space in the vg0 partition for my partition schema.

Long term solution

Investigate on the docker overlay volume size culprits

Quick solution

Reconfigure docker so that the data-root folder points to a directory in a bigger volume (like the one used to store asset, usually several TBs large)

Moving the Docker overlay2 volume to another directory within the same filesystem involves a process that ensures all your Docker data is safely transferred to the new location without data loss. Here's how you can do it:

Step 1: Stop Docker Service

First, you need to stop Docker to ensure that no containers are running and no changes are being made to the data during the move. You can stop Docker using the following command:

sudo systemctl stop docker

Step 2: Copy the Docker Data Directory

Next, copy the entire Docker data directory (/var/lib/docker by default) to the new location. You can use the rsync command for this purpose as it's efficient and can resume interrupted transfers. Replace /new/docker/data/location with the path to where you want to move the Docker data.

sudo rsync -avzh /var/lib/docker/ /new/docker/data/location/

Step 3: Update Docker's Storage Location

After copying the data, you need to tell Docker to use the new location for its data. This is done by editing or creating the Docker daemon configuration file, usually found at /etc/docker/daemon.json. If this file does not exist, you will create it; if it does, add or modify the data-root attribute.

{
  "data-root": "/new/docker/data/location"
}

Make sure to replace /new/docker/data/location with the actual new path of your Docker data.

Step 4: Restart Docker Service

Now, restart Docker to apply the changes. Docker will now use the new directory for all its data, including overlay2.

sudo systemctl start docker

Step 5: Verify and Clean Up

After Docker starts, verify that your containers and images are correctly recognized and that everything operates as expected. If all is well, you can remove the old Docker data directory to free up space on your system. Ensure that everything is working correctly before doing this, as this step is irreversible.

sudo rm -rf /var/lib/docker/

Important Considerations

• Backups: Before proceeding with the move, it's highly recommended to backup your Docker data to avoid any loss.
• Docker Version: Ensure the Docker version is the same before and after the move to avoid compatibility issues.
• Disk Space: Ensure the destination directory has enough space to accommodate the Docker data.
• Permissions: Keep an eye on permissions and ownership of the new directory to match the original. Docker needs specific permissions to manage its data correctly.

By following these steps, you should be able to move the Docker overlay2 volume to a new directory on the same filesystem without losing your Docker containers and images.