docker-volume-netshare
docker-volume-netshare copied to clipboard
ContainX
netshare volume plugin not working in "swarm mode" v1.12.0-rc3 with docker volume create, docker service create --mount
Output of docker version:
Client:
Version: 1.12.0-rc3
API version: 1.24
Go version: go1.6.2
Git commit: 91e29e8
Built: Sat Jul 2 00:17:11 2016
OS/Arch: darwin/amd64
Server:
Version: 1.12.0-rc3
API version: 1.24
Go version: go1.6.2
Git commit: 91e29e8
Built: Sat Jul 2 00:17:11 2016
OS/Arch: linux/amd64
Output of docker info:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 1
Server Version: 1.12.0-rc3
Storage Driver: aufs
Root Dir: /mnt/sda1/var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 1
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local nfs
Network: overlay bridge host null
Swarm: active
NodeID: 91wk9arjzrqnkzdvigu4wzcii
IsManager: Yes
Managers: 1
Nodes: 4
CACertHash: sha256:8e03c7e964a135ab6200346e7de585139e08b725d4fe62fd7dc0558104b838a6
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.14-boot2docker
Operating System: Boot2Docker 1.12.0-rc3 (TCL 7.1); HEAD : 8d9ee9f - Sat Jul 2 05:02:44 UTC 2016
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 492.7 MiB
Name: swmaster-1
ID: I5SS:M4L6:ZKWZ:W37G:FJ24:VLMF:BCHA:LRAF:GVQ3:C2MR:INFO:WE3Y
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 35
Goroutines: 144
System Time: 2016-07-09T18:57:45.749715718Z
EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
provider=virtualbox
Insecure Registries:
127.0.0.0/8
Additional environment details:
docker-machine create -d virtualbox \
--virtualbox-boot2docker-url https://github.com/boot2docker/boot2docker/releases/download/v1.12.0-rc3/boot2docker.iso \
--virtualbox-memory "512" swmaster-1
docker-machine create -d virtualbox \
--virtualbox-boot2docker-url https://github.com/boot2docker/boot2docker/releases/download/v1.12.0-rc3/boot2docker.iso \
--virtualbox-memory "512" swnode-1
docker $(docker-machine config swmaster-1) swarm init --listen-addr $(docker-machine ip swmaster-1):2377
docker $(docker-machine config swnode-1) swarm join $(docker-machine ip swmaster-1):2377 --listen-addr $(docker-machine ip swnode-1):2377
netshare is running in every host, including master https://github.com/ContainX/docker-volume-netshare/releases/download/v0.18/docker-volume-netshare_0.18_linux_amd64-bin:
sudo /var/lib/boot2docker/bin/docker-volume-netshare nfs -v 3 --basedir=/mnt
netshare is working with docker run -v
docker run -it --volume-driver=nfs -v 192.168.99.1/Volumes/HDD/tmp:/mnt --name test1 alpine /bin/ls /mnt
## List all files in 192.168.99.1/Volumes/HDD/tmp OK
netshare is not working with docker volume create + docker service create
docker volume create -d nfs --name /192.168.99.1/Volumes/HDD/tmp
docker service create --endpoint-mode dnsrr --mount type=volume,source=192.168.99.1/Volumes/HDD/tmp,target=/mnt,volume-driver=nfs --name test10 alpine ping uol.com.br
Nothing is mounted ...
docker service inspect test10 shows:
"Mounts": [
{
"Type": "volume",
"Source": "192.168.99.1/Volumes/HDD/tmp",
"Target": "/mnt",
"VolumeOptions": {
"DriverConfig": {
"Name": "nfs"
}
}
}
]
docker service tasks test10 shows:
ID NAME SERVICE IMAGE LAST STATE DESIRED STATE NODE
3li5cdzauffoqhrlktpu72qhx test10.1 test10 alpine Running 20 minutes Running swnode-1
Inside swnode-1 docker ps -a shows:
463fd2f94510 alpine:latest "ping uol.com.br" 7 minutes ago Up 7 minutes test10.1.3li5cdzauffoqhrlktpu72qhx
And finally, docker inspect 463fd2f94510 shows:
Mounts": [
{
"Name": "48fc07f3be24cc706598e3082316cb547fdd93d24df3b97c78b6c8f4862edc52",
"Source": "/mnt/sda1/var/lib/docker/volumes/48fc07f3be24cc706598e3082316cb547fdd93d24df3b97c78b6c8f4862edc52/_data",
"Destination": "/mnt:ro,nocopy",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
}
],
Inside the container test10.1 a folder named /mnt:ro,nocopy is created.
$ docker exec 463fd2f94510 /bin/ls /
bin
dev
etc
home
lib
linuxrc
media
mnt
mnt:ro,nocopy
proc
root
run
sbin
srv
sys
tmp
usr
var
If I change the target= to any other folder, I still getting 'folder_name:ro,nocopy' created inside the container.
netshare is not working with docker volume create + docker service create
docker volume create -d nfs --name /192.168.99.1/Volumes/HDD/tmp
What is the output of docker volume ls afterwards?
Could you try
docker volume create -d nfs --name somename -o share=192.168.99.1/Volumes/HDD/tmp
Havn't had time to play with 1.12-rc yet, however I found the following new example in the 1.12 docs
Could you try the follwoing example from the doc for creation of the volume?
$ docker volume create --driver local --opt type=nfs --opt o=addr=192.168.1.1,rw --opt device=:/path/to/dir --name foo
But I'd say that there is a problem within the new docker service command, because it seems to change the driver from nfs to local (according to docker inspect).
Further the following output from docker service inspect looks strange:
"VolumeOptions": {
"DriverConfig": {
"Name": "nfs"
}
I wouldn't expect to see "Name": "nfs" rather I'd expect maybe "Type": "nfs". But this is just guesing...
BTW: where did you found a detailed reference to the --mountOption of docker service create?
Hi, It seems docker team have done some work in 1.12.0-rc4 #24484 I will give it a try and let you know. Do you know if ContainX is working with the experimental "docker plugin" command ?
For the --mount docs, I found this issue #24277 There is another issue (open or closed, i dont know) at https://github.com/docker/docker
With docker v1.12.1 in swarm mode I get:
$ docker volume create --driver=local --opt=type=nfs --opt=o=addr=192.168.1.1,rw --opt device=:/vol/vol_docker_registry_mirror --name=registry-mirror
registry-mirror
$ docker volume ls
DRIVER VOLUME NAME
local registry-mirror
$ docker volume inspect registry-mirror
[
{
"Name": "registry-mirror",
"Driver": "local",
"Mountpoint": "/var/lib/docker/volumes/registry-mirror/_data",
"Labels": {},
"Scope": "local"
}
]
...at which point docker run -it --rm -v registry-mirror:/data ubuntu /bin/bash does work for me.
However, I'm mildly concerned. When I ran this the first time, additional volumes (previously invisible) suddenly appeared. The volumes were my previous attempts to get this to work. :-/
I was able to get a service to work, using that volume command. But I had to run the volume command by hand on every node in the swarm. It looks like it tried to auto-copy the volumes over, but it failed to work. :-/
The service command (for posterity):
$ docker service create --replicas=3 --mount=type=volume,src=registry-mirror,dst=/var/lib/registry --name=mirror -p 5010:443 registry:2 serve /var/lib/registry/config.yml
@docwhat I can confirm that it works with your solution.
Docker version 1.12.1, build f1e1b83
Core-OS version 1164.1.0
# first create the volume on all nodes where the container is to be ran (core-03 in this example)
#
# if I don't create the volumes on each node the service will start without any error messages but the volume won't be mounted inside the container
docker volume create --driver=local --opt=type=nfs --opt=o=addr=10.10.10.66,rw --opt device=:/storage/mysql --name=mysql-staging
# start the service on core-03
docker service create --mount=type=volume,src=mysql-staging,dst=/mysql --name=nginx --constraint 'node.hostname == core-03' -p 8080:80 nginx
# verify that the volume is mounted in the container ran on core-03
docker exec a5f4b52bece4 mount | grep storage
:/storage/mysql on /mysql type nfs (rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.10.66,mountvers=3,mountproto=tcp,local_lock=none,addr=10.10.10.66)
docker exec a5f4b52bece4 ls /mysql/data
auto.cnf
ib_buffer_pool
ib_logfile0
ib_logfile1
ibdata1
ibtmp1
mysql
performance_schema
sys
~~Everything works without docker-volume-netshare even beeing started. The only thing not working without the daemon started is docker volume ls, I get this error message:~~
list nfs: Post http://%2Frun%2Fdocker%2Fplugins%2Fnfs.sock/VolumeDriver.List: dial unix /run/docker/plugins/nfs.sock: connect: connection refused
DRIVER VOLUME NAME
local mysql-staging
~~But everything else is as said working.~~
EDIT:
After removing the file /run/docker/plugins/nfs.sock everything works.
So the question now is, why do I need docker-volume-netshare when everything seems to work out of the box?
Could someone clarify if it is required to create the volumes on every node in the swarm in order to get this to work? currently I am getting nfs volumes in the manager, but on the rest of the nodes I just get driver=local, seems like the swarm is distributing the volumes across the nodes but does not understand that the volumes are using the netshare plugin. Furthermore, @simonwahlgren seems to manage to mount nfs using the local driver, is there any disadvantage using this approach? does it also require to create the volumes in every node?
In Docker version 1.12.3, build 6b644ec and docker-volume-netshare_0.20:
You must create nfs volume on every node, otherwise the volume will be empty.
leader,node2,node3
#run only in leader
docker volume create -d nfs --name nfshare -o share=${NFSHARE_SERVER}:/share
docker service create \
--mount type=volume,src=nfshare,dst=/share \
--publish 8080:80 \
--replicas=10 \
--name nginx nginx
Then ssh to node2, node3 and enter nginx, the /share is empty.
@o3o3o As far as I know (I learnt this recently), this is the way volumes are supposed to work in swarm. The only supported way is to create the volume when creating the service: https://docs.docker.com/engine/reference/commandline/service_create/
In my opinion not a good approach, but I hope this is just because swarm node is quite new and they have not figured out everything just yet.
Are volumes suppose to persist across docker-volume-netshare service restarts?
I setup my volumes with the docker volumes command, but when restarting the docker-volume-netshare service they disappear. This worries me that the containers will lose their volumes if the host reboots.
There's absolutely no way to define the volume when creating the service? I can get single nodes to work with the volumes, but not all nodes at the same time.
@blandman that's my predicament, too. I don't want to have to manually create the volume on each host.
same problem here:
root@docker3:/opt# docker volume create --driver nfs --name netsharenfs -o share=192.168.253.100:/
netsharenfs
root@docker3:/opt# docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
0zizhqemxr3fmhr949qqg94ly docker1 Ready Active
88qpfbyudpeqzrfabvhptnz36 docker2 Ready Active
wx1fj0df7mdkihr5855xlfdcx * docker3 Ready Active Leader
root@docker3:/opt# docker service create --mode=global --mount src=netsharenfs,dst=/nfs --name nfstest phusion/baseimage
fup2srz49c7p6uw9vu0bf8w82
Since --detach=false was not specified, tasks will be created in the background.
In a future release, --detach=false will become the default.
root@docker3:/opt# docker volume inspect netsharenfs
[
{
"Driver": "nfs",
"Labels": {},
"Mountpoint": "/var/lib/docker-volumes/netshare/nfs/netsharenfs",
"Name": "netsharenfs",
"Options": {
"share": "192.168.253.100:/"
},
"Scope": "global"
}
]
root@docker1:/opt# docker volume inspect netsharenfs
[
{
"Driver": "local",
"Labels": null,
"Mountpoint": "/var/lib/docker/volumes/netsharenfs/_data",
"Name": "netsharenfs",
"Options": {},
"Scope": "local"
}
]
root@docker2:/opt# docker volume inspect netsharenfs
[
{
"Driver": "local",
"Labels": null,
"Mountpoint": "/var/lib/docker/volumes/netsharenfs/_data",
"Name": "netsharenfs",
"Options": {},
"Scope": "local"
}
]
root@docker1:/opt# service docker-volume-netshare status
● docker-volume-netshare.service - LSB: Init for docker-volume-netshare
Loaded: loaded (/etc/init.d/docker-volume-netshare; bad; vendor preset: enabled)
Active: active (running) since Mon 2017-08-28 13:10:35 CEST; 1h 18min ago
Docs: man:systemd-sysv-generator(8)
Process: 6607 ExecStart=/etc/init.d/docker-volume-netshare start (code=exited, status=0/SUCCESS)
Tasks: 5
Memory: 3.2M
CPU: 55.421s
CGroup: /system.slice/docker-volume-netshare.service
└─6618 /usr/bin/docker-volume-netshare nfs
Aug 28 13:10:35 docker1 systemd[1]: Starting LSB: Init for docker-volume-netshare...
Aug 28 13:10:35 docker1 docker-volume-netshare[6607]: * Starting Docker-Volume-Netshare: docker-volume-netshare
Aug 28 13:10:35 docker1 docker-volume-netshare[6607]: ...done.
Aug 28 13:10:35 docker1 systemd[1]: Started LSB: Init for docker-volume-netshare.
Aug 28 13:11:46 docker1 systemd[1]: Started LSB: Init for docker-volume-netshare.
So nothing but the Name of the volume is replicated to the other worker nodes. I've just had the same problem with local driver and type NFS here: https://github.com/moby/moby/issues/34556
Unfortunately, the netshare-driver doesn't make this even besser.
If I can help with additional infos or tests, please contact me. I would like to help making this "swarm-ready".
//edited: However, as swarms are mostly deployed via the yml-Files, the Volumes WITH the parameters get running on all nodes. @simonwahlgren So the difference seems that the local driver uses NFSv3 and this here NFSv4 (by default)
Looks to me thats this is maybe a general Swarm problem, because for me even original local volumes lost there configuration when they get applied to worker nodes
This problem has some time, so most likely won't be able to get much help, but nonetheless, I'll give it a shot, maybe someone can help...
I'm trying to use the plugin with Docker Swarm. Naturally, I don't want to have to manually create the volumes on every node, that's unthinkable, so I'm using a named volume, which means that if it doesn't exist, it's created, if it does, it re-uses it.
The command that I'm using to create the service is as follows
docker service create --name nuget-server --with-registry-auth \
--log-driver json-file --log-opt max-size=2mb \
--constraint 'node.role==worker' \
--mount 'type=volume,src=baget,dst=/var/baget,volume-driver=nfs,volume-opt=share=1.1.1.1:/mnt/volume_3/nfs/baget' \
--env ApiKey=some-random-key \
--env Storage__Type=FileSystem \
--env Storage__Path=/var/baget/packages \
--env Database__Type=Sqlite \
--env Search__Type=Database \
baget:latest
That seems to work, as it follows the parameters as when manually creating the volume. And it looks ok as well.
[
{
"CreatedAt": "0001-01-01T00:00:00Z",
"Driver": "nfs",
"Labels": {},
"Mountpoint": "/var/lib/docker-volumes/netshare/nfs/baget",
"Name": "baget",
"Options": {
"share": "1.1.1.1:/mnt/volume_3/nfs/baget"
},
"Scope": "local"
}
]
So, it all looked good, until I try to write to the folder mapped to the volume, where I get an unauthorized error. Can anyone shed some light on this? Maybe I'm not doing the volume creation properly via service create?
Cheers
