Implement import & export functionality for volumes

[rcdailey]

I would like the ability to import and export volumes in Docker. I will explain each of these scenarios below. The motivation for this is that sometimes I like to be able to easily transition between bind mounts on the host and volumes managed by docker. If I previously started a container with a bind mount (which requires a directory on the host), I want to easily create a new volume with the contents of that directory, and run the container next time using a volume instead of a bind mount. The vice-versa of this scenario should also be supported. At the end of the day, the container doesn't know the difference and still sees the same data, this is just a management task for the host machine.

I also feel docker doing this is safer since it already has root privileges, and sudo won't be needed to copy in/out of /var/lib/docker which at the moment is locked down for even viewing.

Import Volumes

Use case: I have a directory that I want to recursively box into a docker volume. The end result of this operation should be:

docker volume create newvolume
cp -r /some/directory /var/lib/docker/volumes/newvolume/_data

Proposed docker command line usage for such a feature:

docker volume import /some/directory newvolume

This recursively copies the contents of /some/directory into a new volume named newvolume. Permissions should be retained during the copy (owners & permission bits). This is really just sugar coating over mkdir and cp, as shown in my earlier example.

Export Volumes

Use case: I have a volume that I want to convert to a directory, where that directory contains an exact recursive replica of the data in the volume. The end result of this operation should effectively be:

cp -r /var/lib/docker/volumes/existingvolume/_data/* /some/export/path

Example docker command line for such feature:

docker volume export existingvolume /some/export/path

pre & post conditions:

• Destination directory must exist
• The destination directory must be empty
• The destination directory will receive the contents of the volume (recursively)
• The destination directory contents will retain the permission bits & owner/group that was there in the volume itself

[cpuguy83]

import and export are pretty ambiguously used here. I would expect import and export to deal with actual volumes instead of just the data of a volume.

Just copying data from some directory to another is also something that is easily performed with a docker run, e.g. docker run -v /some/directory:/tmp/src -v someVolume:/tmp/dst busybox cp -r /tmp/src/* /tmp/dst/

And this could be optimized with other tools like rsync, or some storage specific optimization, which is something that a command like export and import could possibly optimize for.

[rcdailey]

Granted I don't know the ins and outs of docker, or its cli design principles, but really using run for this just feels hacky. It's more of a workaround. I prefer a polished and simple solution like with import and export. Opinions on naming ambiguity aside, I think it's much simpler and more trivial than what you've suggested. I don't like using run for this at all.

I think we could have a very broad debate on what "import" and "export" mean in various contexts, but I think part of this feature is about picking one and defining it clearly: Later you could expand these to import/export volumes to tarballs or something, to make it a little closer to what you're probably thinking (maybe some backup mechanism?). But I think at the end of the day, since volumes really do deal in data, the terms aren't that ambiguous.

Don't forget that part of this is about keeping things simple.

[thaJeztah]

For reference; earlier discussions around this feature (or similar);

• https://github.com/moby/moby/issues/18718 Add support for importing or exporting from volumes
• https://github.com/moby/moby/issues/25245 How to copy files into a named volume without an attached container?
• https://github.com/moby/moby/issues/31417 how can i export an docker volume to a zip or tar file
• also https://stackoverflow.com/questions/21597463/how-to-port-data-only-volumes-from-one-host-to-another/23778599#23778599
• https://github.com/moby/moby/issues/32263
• https://github.com/docker/cli/issues/1436

[trajano]

Is there a separate feature request to simply copy one file from the volume? Just one way, not bi-directional, which can be added later.

I'm looking for something like

docker volume cp <volume>:/tmp/file file.log

That implements something like

C=$( docker run -d -v volume:/mnt alpine )
docker cp $C:/mnt/tmp/file file.log
docker rm -f $C >/dev/null

In a single command.

For now I created a bash function that overrides by docker command to add the functionality as follows. It's janky but works. I also provided a files function which lists files in the volume.

  elif [ "${ARGS[0]}" == 'volume' -a  "${ARGS[1]}" == 'files' ]
  then
    command docker run --rm -v ${ARGS[2]}:/mnt alpine //bin/sh -c "find /mnt -print | sed 's/^\/mnt//'"
  elif [ "${ARGS[0]}" == 'volume' -a  "${ARGS[1]}" == 'cp' ]
  then
    local C=$( command docker run -d -v ${ARGS[2]}:/mnt alpine )
    command docker cp $C:/mnt${ARGS[3]} ${ARGS[4]}
    command docker rm -f $C > /dev/null

[spkane]

@thaJeztah Has there been any additional discussion about this in the last few years?

It seems silly at this point in Docker's lifecycle that there is no built-in command to backup and distribute data volumes from the CLI.

[thaJeztah]

@spkane it's not currently being worked on, but it continues to be a topic that comes up once in a while, but there's still parts that needs to be looked at closely, especially if a volume is "in use", saving the data is not always trivial (files may be modified during the export operation).

However, if you run Docker Desktop, you can install the "Volumes Backup & Share" extension which provides this functionality https://hub.docker.com/extensions/docker/volumes-backup-extension

I should warn there (these are parts that need further discussion) it also allows pushing a volume as an image (which can be a "foot-gun" if you accidentally push your sensitive data as a public image, in addition to images not being efficient for preserving large volumes where only "some" files changed) 😅

[spkane]

@thaJeztah Thank you for the quick response. I am aware of the extension and have managed to cobble together a reasonably simple tar command (from one of your posts years ago) to handle creating a tarball that can then be un-archived into a new data volume for my use case, which was to make it easy to distribute a pre-populated DB for some container classes that I teach.

I can see why this is tricky, although it might be possible to use the container pause/freeze features to prevent writing during the export or require the data volume to be unused. At least in my case, I would be okay with not uploading it to a registry, but I do wish that I could export and import the data volume to a local file directly instead of needing to lean on tar.