Correct way to mount NFS share

[BurningTheSky]

Question

I want to be able to mount a rw share into a container, I have tried some Docker specific methods but they don't appear to work. Is there a way to do it with VIC?

See also

[burningsky@engal ~]$ export NFS_VOL_NAME=nfs NFS_LOCAL_MNT=/mnt/nfs NFS_SERVER=192.168.0.14 NFS_SHARE=/data/nfs NFS_OPTS=vers=4,soft docker --tls run --mount "src=$NFS_VOL_NAME,dst=$NFS_LOCAL_MNT,volume-opt=device=:$NFS_SHARE,"volume-opt=o=addr=$NFS_SERVER,$NFS_OPTS",type=volume,volume-driver=local,volume-opt=type=nfs" busybox ls $NFS_LOCAL_MNT Unable to find image 'busybox:latest' locally latest: Pulling from library/busybox bdbbaa22dec6: Pull complete a3ed95caeb02: Pull complete Digest: sha256:edafc0a0fb057813850d1ba44014914ca02d671ae247107ca70c94db686e7de6 Status: Downloaded newer image for library/busybox:latest ERRO[0193] error waiting for container: Error response from daemon: Server error from portlayer: ContainerWaitHandler(3c6d0cdf98e44764bc982ec25133c16aaf4ff30ea151165a4d430eb6796fd769) Error: context deadline exceeded docker: Error response from daemon: Server error from portlayer: unable to wait for process launch status: Post https://vcenter.domain.local/sdk: context deadline exceeded. [burningsky@engal ~]$ docker --tls volume create --driver local --opt type=nfs --opt o=addr=$NFS_SERVER,$NFS_OPTS --opt device=:$NFS_SHARE $NFS_VOL_NAME Error response from daemon: bad driver value - device is not a supported option

[YanzhaoLi]

Just as this doc(https://vmware.github.io/vic-product/assets/files/html/1.5/vic_vsphere_admin/backup_volumes.html) shows, you should first create a NFS-type volumeStore, and then use this volumeStore to create docker volume. And this is the only way to create NFS Shared Volumes in VIC

# create or configure vch with the volumestore parameter
--volume-store nfs://10.118.68.164/mnt/nfs-vol?uid=0&gid=0&proto=tcp:shared
# create nfs volume
docker volume create --opt VolumeStore=shared myshared
# use nfs volume
docker run -v myshared:/share xx

[BurningTheSky]

Thanks, so I would need to redeploy the vch, there's no way to reconfigure a running one?

[YanzhaoLi]

It's ok to reconfigure the VCH. See the "Add Volume Stores" section in this doc: https://vmware.github.io/vic-product/assets/files/html/1.5/vic_vsphere_admin/configure_vch.html

[BurningTheSky]

I'm not sure what I've done wrong but the mount options appear to be wrong?

C:\Users\burningsky>vic-machine-windows configure --target vcenter.domain.local --user [email protected] --password password --thumbprint xxxxxxxxxxxxxx --id virtual-container-host --volume-store VM SAS 15K/DockerFolder:default --volume-store VM SAS 15K/DockerFolder:DockerVolume --volume-store nfs://192.168.0.14/Multimedia?uid=0&gid=0&proto=tcp:NFSMultimedia time="2020-02-24T15:07:32Z" level=error msg=-------------------- time="2020-02-24T15:07:32Z" level=error msg="vic-machine-windows configure failed: volume store input must be in format <datastore url w/ path>:label or nfs:///?:\n" 'gid' is not recognized as an internal or external command, operable program or batch file. 'proto' is not recognized as an internal or external command, operable program or batch file.

[YanzhaoLi]

It seems the windows shell could not parse special characters. Please try to use quote like --volume-store "nfs://192.168.0.14/Multimedia?uid=0&gid=0&proto=tcp:NFSMultimedia"

[BurningTheSky]

I did give that a try before but no luck:

C:\Users\burningsky>vic-machine-windows configure --target vcenter.domain.local --user [email protected] --password password --thumbprint xxxxxxxxxxxxxx --id virtual-container-host --volume-store VM SAS 15K/DockerFolder:default --volume-store VM SAS 15K/DockerFolder:DockerVolume --volume-store "nfs://192.168.0.14/Multimedia?uid=0&gid=0&proto=tcp:NFSMultimedia" time="2020-02-24T18:18:31Z" level=error msg=-------------------- time="2020-02-24T18:18:31Z" level=error msg="vic-machine-windows configure failed: volume store input must be in format <datastore url w/ path>:label or nfs:///?:\n"

[BurningTheSky]

So I tried from linux but got a similar response: [burningsky@engal ~]$ vic-machine-linux configure --target vcenter.domain.local --user [email protected] --password password --thumbprint xxxxxx --id virtual-container-host --volume-store VM SAS 15K/DockerFolder:default --volume-store VM SAS 15K/DockerFolder:DockerVolume --volume-store nfs://192.168.0.14/Multimedia?uid=0&gid=0&proto=tcp:NFSMultimedia [1] 19971 [2] 19972 [burningsky@engal ~]$ ERRO[0000] -------------------- ERRO[0000] vic-machine-linux configure failed: volume store input must be in format <datastore url w/ path>:label or nfs:///?:

Then I tried to get the current config but it failed on the ID, but that is the name of the VCH, is there another ID other than its name and if so how do I find it? [burningsky@engal ~]$ vic-machine-linux inspect config --target vcenter.domain.local --user [email protected] --password password --thumbprint xxxxxx --id virtual-container-host INFO[0000] ### Inspecting VCH #### INFO[0000] Validating target ERRO[0000] Failed to get Virtual Container Host virtual-container-host ERRO[0000] id "virtual-container-host" could not be found ERRO[0000] -------------------- ERRO[0000] vic-machine-linux inspect config failed: inspect failed

[YanzhaoLi]

1. you can use --name VCH_NAME. And also to find the id, you can use vic-machine-linux ls
2. In linux, please use single quote --volume-store 'nfs://192.168.0.14/Multimedia?uid=0&gid=0&proto=tcp:NFSMultimedia'

[BurningTheSky]

Thanks, the single quotes worked but now I'm running into a timeout error, any idea what's causing that?

[burningsky@engal ~]$ vic-machine-linux configure --target vcenter.domain.local --user [email protected] --password password --thumbprint xxxxx --id vm-298 --volume-store 'VM SAS 15K/DockerFolder:default' --volume-store 'VM SAS 15K/DockerFolder:DockerVolume' --volume-store 'nfs://192.168.0.14/Multimedia?uid=0&gid=0&proto=tcp:NFSMultimedia' INFO[0000] ### Configuring VCH #### INFO[0000] Validating target INFO[0000] INFO[0000] VCH ID: VirtualMachine:vm-298 INFO[0000] Creating directory [VM SAS 15K] virtual-container-host INFO[0000] Datastore path is [VM SAS 15K] virtual-container-host INFO[0001] No certificate regeneration requested. No new certificates provided. Certificates left unchanged. INFO[0001] Validating supplied configuration INFO[0001] Configuring static IP for additional networks using port group "Management" INFO[0001] Network configuration OK on "Containers" INFO[0001] Network configuration OK on "Management" INFO[0001] vCenter settings check OK INFO[0001] Firewall status: ENABLED on "/domain/host/esxi1.domain.local/esxi1.domain.local" INFO[0001] Firewall configuration OK on hosts: INFO[0001] "/domain/host/esxi1.domain.local/esxi1.domain.local" INFO[0001] vCenter settings check OK INFO[0002] License check OK on hosts: INFO[0002] "/domain/host/esxi1.domain.local/esxi1.domain.local" INFO[0002] DRS check SKIPPED - target is standalone host WARN[0003] Guest tools unavailable, resorting to power off - sessions will be left open INFO[0005] Creating directory [VM SAS 15K] DockerFolder INFO[0005] Datastore path is [VM SAS 15K] DockerFolder INFO[0005] Creating directory [VM SAS 15K] DockerFolder INFO[0005] Datastore path is [VM SAS 15K] DockerFolder INFO[0005] Setting VM configuration INFO[0007] Waiting for IP information INFO[0007] Waiting for major appliance components to launch INFO[0014] Obtained IP address for client interface: "192.168.0.202" INFO[0014] Checking VCH connectivity with vSphere target INFO[0015] vSphere API Test: https://vcenter.domain.local vSphere API target responds as expected ERRO[0180] vic/lib/install/management.(*Dispatcher).CheckDockerAPI: CheckServiceReady during update error: context deadline exceeded vic/lib/install/management.(*Dispatcher).update:348 CheckServiceReady during update vic/cmd/vic-machine/configure.(*Configure).Run:368 vic-machine-linux vic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux INFO[0180] Docker API endpoint check failed: context deadline exceeded ERRO[0180] vic/lib/install/management.(*Dispatcher).update: CheckServiceReady during update error: context deadline exceeded vic/lib/install/management.(*Dispatcher).update:348 CheckServiceReady during update vic/cmd/vic-machine/configure.(*Configure).Run:368 vic-machine-linux vic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux INFO[0180] API may be slow to start - please retry with increased timeout using --timeout ERRO[0180] Failed to configure: Upgrading VCH exceeded time limit of 3m0s. Please increase the timeout using --timeout to accommodate for a busy vSphere target INFO[0180] Rolling back configure INFO[0180] Reverting to snapshot reconfigure for 21221 ERRO[0180] vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded vic/cmd/vic-machine/configure.(*Configure).Run:368 vic-machine-linux vic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux ERRO[0180] vic/pkg/vsphere/tasks.WaitForResultAndRetryIf: vic-machine-linux error: context deadline exceeded vic/cmd/vic-machine/configure.(*Configure).Run:368 vic-machine-linux vic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux ERRO[0180] vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded vic/cmd/vic-machine/configure.(*Configure).Run:368 vic-machine-linux vic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux ERRO[0180] Failed to revert appliance to snapshot: Failed to roll back upgrade: context deadline exceeded. INFO[0180] Collecting ed2845f0-de50-46d3-b1fa-91f005e5b487 vpxd.log ERRO[0180] vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded vic/cmd/vic-machine/configure.(*Configure).Run:368 vic-machine-linux vic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux ERRO[0180] Failed to collect ed2845f0-de50-46d3-b1fa-91f005e5b487 vpxd.log: Post https://vcenter.domain.local/sdk: context deadline exceeded WARN[0180] No log data for ed2845f0-de50-46d3-b1fa-91f005e5b487 vpxd.log ERRO[0180] vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded vic/cmd/vic-machine/configure.(*Configure).Run:368 vic-machine-linux vic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux ERRO[0180] vic/pkg/vsphere/tasks.WaitForResultAndRetryIf: vic-machine-linux error: context deadline exceeded vic/cmd/vic-machine/configure.(*Configure).Run:368 vic-machine-linux vic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux ERRO[0180] vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded vic/cmd/vic-machine/configure.(*Configure).Run:368 vic-machine-linux vic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux ERRO[0180] Failed to reset UpdateInProgress ERRO[0180] context deadline exceeded ERRO[0180] vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded vic/cmd/vic-machine/configure.(*Configure).Run:368 vic-machine-linux vic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux ERRO[0180] vic/cmd/vic-machine/configure.(*Configure).Run.func2: vic-machine-linux error: context deadline exceeded vic/cmd/vic-machine/configure.(*Configure).Run:368 vic-machine-linux vic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux ERRO[0180] vic/cmd/vic-machine/configure.(*Configure).Run.func2: vic-machine-linux error: context deadline exceeded vic/cmd/vic-machine/configure.(*Configure).Run:368 vic-machine-linux vic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux ERRO[0180] -------------------- ERRO[0180] vic-machine-linux configure failed: Configure timed out: use --timeout to add more time

[BurningTheSky]

Shall I just add more timeout and see if it completes? How much should I add?

[BurningTheSky]

If I check the config of the vch is is showing the new datastore in there, and the vcenter is reporting constant reconfigurations on the container VMs now... Something strange is going on

[YanzhaoLi]

It is probably due to failing to connect the nfsserver.

• Please make sure the nfs-server work well, which means you can mount it.
• Check the vch's tether.debug. Download it from VC: datastore->vch_name->tether.debug. There might be errors like error occurred while attempting to mount volumestore (NFSMultimedia). If not, we should do further debugging.

[BurningTheSky]

I can't see anything NFS related but the error loop in there has been going constantly from then until now: 2020/02/26 08:25:59 unknown command: "Time_Synchronize" 2020/02/26 08:25:59 Message: Unable to send a message over the communication channel 0 all stderr redirected to debug logall stdout redirected to debug logFeb 26 2020 08:26:23.172Z INFO Registering tether extension Toolbox Feb 26 2020 08:26:23.173Z INFO Registering tether extension logrotate Feb 26 2020 08:26:23.174Z INFO opening var/log/vic/init.log for debug log Feb 26 2020 08:26:23.175Z INFO Started reaping child processes Feb 26 2020 08:26:23.176Z INFO unmounting /etc/hosts Feb 26 2020 08:26:23.177Z INFO bind-mounting /.tether/etc/hosts on /etc/hosts Feb 26 2020 08:26:23.179Z INFO Starting extension Toolbox Feb 26 2020 08:26:23.180Z INFO toolbox: waiting for initialization Feb 26 2020 08:26:23.181Z INFO Launching vch-init pprof server on 127.0.0.1:6060 2020/02/26 08:26:23 dispatching power op "OS_PowerOn" Feb 26 2020 08:26:23.187Z INFO toolbox: service is ready (power on event received) Feb 26 2020 08:26:23.188Z INFO Starting extension logrotate Feb 26 2020 08:26:23.189Z INFO Loading main configuration Feb 26 2020 08:26:23.219Z INFO unmounting /etc/hostname Feb 26 2020 08:26:23.220Z INFO bind-mounting /.tether/etc/hostname on /etc/hostname Feb 26 2020 08:26:23.223Z INFO unmounting /etc/hosts Feb 26 2020 08:26:23.225Z INFO bind-mounting /.tether/etc/hosts on /etc/hosts Feb 26 2020 08:26:23.227Z INFO Renaming link eth1 to bridge Feb 26 2020 08:26:23.231Z INFO successfully applied config for network bridge Feb 26 2020 08:26:23.232Z INFO Renaming link eth0 to client Feb 26 2020 08:26:23.242Z INFO setting ip address 192.168.0.202/24 for link client Feb 26 2020 08:26:23.245Z WARN ignoring incomplete line "127.0.1.1" Feb 26 2020 08:26:23.246Z INFO unmounting /etc/hosts Feb 26 2020 08:26:23.248Z INFO bind-mounting /.tether/etc/hosts on /etc/hosts Feb 26 2020 08:26:23.250Z INFO Added nameservers: [192.168.0.87] Feb 26 2020 08:26:23.251Z INFO unmounting /etc/resolv.conf Feb 26 2020 08:26:23.252Z INFO bind-mounting /.tether/etc/resolv.conf on /etc/resolv.conf Feb 26 2020 08:26:23.254Z INFO successfully applied config for network client Feb 26 2020 08:26:23.256Z INFO setting ip address 192.168.0.202/24 for link client Feb 26 2020 08:26:23.257Z WARN address 192.168.0.202/24 already set on interface client Feb 26 2020 08:26:23.258Z INFO unmounting /etc/hosts Feb 26 2020 08:26:23.260Z INFO bind-mounting /.tether/etc/hosts on /etc/hosts Feb 26 2020 08:26:23.262Z INFO Added nameservers: [192.168.0.87] Feb 26 2020 08:26:23.264Z INFO unmounting /etc/resolv.conf Feb 26 2020 08:26:23.266Z INFO bind-mounting /.tether/etc/resolv.conf on /etc/resolv.conf Feb 26 2020 08:26:23.268Z INFO successfully applied config for network management Feb 26 2020 08:26:23.269Z INFO Aliasing link client to public Feb 26 2020 08:26:23.271Z INFO setting ip address 192.168.0.202/24 for link client Feb 26 2020 08:26:23.272Z WARN address 192.168.0.202/24 already set on interface client Feb 26 2020 08:26:23.274Z INFO Setting default gateway to 192.168.0.1 Feb 26 2020 08:26:23.275Z INFO updated default route to public interface, gateway: 192.168.0.1 Feb 26 2020 08:26:23.276Z INFO unmounting /etc/hosts Feb 26 2020 08:26:23.278Z INFO bind-mounting /.tether/etc/hosts on /etc/hosts Feb 26 2020 08:26:23.280Z INFO Added nameservers: [192.168.0.87] Feb 26 2020 08:26:23.281Z INFO unmounting /etc/resolv.conf Feb 26 2020 08:26:23.282Z INFO bind-mounting /.tether/etc/resolv.conf on /etc/resolv.conf Feb 26 2020 08:26:23.284Z INFO successfully applied config for network public Feb 26 2020 08:26:23.304Z INFO Firewalld running: false Feb 26 2020 08:26:23.309Z INFO opening var/log/vic/vicadmin.log for session logging Feb 26 2020 08:26:23.310Z INFO opening var/log/vic/docker-personality.log for session logging Feb 26 2020 08:26:23.312Z INFO opening var/log/vic/port-layer.log for session logging Feb 26 2020 08:26:23.313Z INFO Launching process for session port-layer Feb 26 2020 08:26:23.314Z INFO Launching process for session vicadmin Feb 26 2020 08:26:23.316Z INFO Launching process for session docker-personality Feb 26 2020 08:26:35.390Z INFO Triggered reload Feb 26 2020 08:26:35.390Z INFO Loading main configuration Feb 26 2020 08:26:35.419Z WARN Session docker-personality already initialized Feb 26 2020 08:26:35.420Z WARN Session port-layer already initialized Feb 26 2020 08:26:35.421Z WARN Session vicadmin already initialized Feb 26 2020 08:26:35.425Z WARN Re-launching process for session port-layer (count: 1) Feb 26 2020 08:26:35.427Z INFO opening var/log/vic/port-layer.log for session logging Feb 26 2020 08:26:45.855Z INFO Triggered reload Feb 26 2020 08:26:45.856Z INFO Loading main configuration Feb 26 2020 08:26:45.905Z WARN Session port-layer already initialized Feb 26 2020 08:26:45.907Z WARN Session vicadmin already initialized Feb 26 2020 08:26:45.908Z WARN Session docker-personality already initialized Feb 26 2020 08:26:45.916Z WARN Re-launching process for session port-layer (count: 2) Feb 26 2020 08:26:45.919Z INFO opening var/log/vic/port-layer.log for session logging . . . . . . . Feb 26 2020 16:45:05.698Z INFO Triggered reload Feb 26 2020 16:45:05.699Z INFO Loading main configuration Feb 26 2020 16:45:05.733Z WARN Session port-layer already initialized Feb 26 2020 16:45:05.734Z WARN Session vicadmin already initialized Feb 26 2020 16:45:05.735Z WARN Session docker-personality already initialized Feb 26 2020 16:45:05.743Z WARN Re-launching process for session port-layer (count: 2822) Feb 26 2020 16:45:05.744Z INFO opening var/log/vic/port-layer.log for session logging

[BurningTheSky]

I've restared the vic and vch containers but the containers are still constantly reconfiguring

[BurningTheSky]

I also can't connect to the docker now: [burningsky@engal ~]$ docker --tls ps -a Cannot connect to the Docker daemon at tcp://192.168.0.202:2376. Is the docker daemon running?

[YanzhaoLi]

That'r really weird.

1. Have you ever tested mounting your nfs-server manually in your local host? Like mount -t nfs 192.168.0.14:/Multimedia LOCALDIR
2. If OK, please collect vch bundle logs for further debug: https://vmware.github.io/vic-product/assets/files/html/1.5/vic_vsphere_admin/log_bundles.html

Thanks.

[BurningTheSky]

I mistyped the path to the NFS, woops. Can I remove it from the config and add the correct path or can you not remove volumes even if they aren't working?

[YanzhaoLi]

em... We didn't support change volume-stores.
Thanks again for your use-case. It seems we should not store the configuration info if we failed to mount the nfs-store.

[BurningTheSky]

Glad to help! End users are good for testing haha. I will redeloy the VCH and see if it goes smoother.

[BurningTheSky]

Making good progress, is there any way to mount an NFS share into the container rather than having to create a shared volume? It would be a pain to have to move all of my data

[YanzhaoLi]

We don't support other ways. But why do you need move all of the data?

[BurningTheSky]

So I want to be able to access data shared with servers as well as the containers. The file structure is /mnt/user/data, but if I put that at as the volume location then it creates the ../volumes/data so I end up with /mnt/user/data/volumes/data. I want the container to have access directly into /mnt/user/data

[m4rkireland]

Any movement on this one?