gluster-kubernetes
gluster-kubernetes copied to clipboard
Error waiting for job 'heketi-storage-copy-job' to complete
As the title says, while using the gk-deploy
script, it ends in failure with the following error: Error waiting for job 'heketi-storage-copy-job' to complete.
Here's the command:
./gk-deploy -g ../../storage_topology.json --admin-key admin --user-key user
And here's the output:
Do you wish to proceed with deployment?
[Y]es, [N]o? [Default: Y]: Y
Using Kubernetes CLI.
Using namespace "default".
Checking for pre-existing resources...
GlusterFS pods ... not found.
deploy-heketi pod ... not found.
heketi pod ... not found.
gluster-s3 pod ... not found.
Creating initial resources ... serviceaccount/heketi-service-account created
clusterrolebinding.rbac.authorization.k8s.io/heketi-sa-view created
clusterrolebinding.rbac.authorization.k8s.io/heketi-sa-view labeled
OK
node/storage-worker-01 labeled
node/storage-worker-02 labeled
node/storage-worker-03 labeled
daemonset.apps/glusterfs created
Waiting for GlusterFS pods to start ... OK
secret/heketi-config-secret created
secret/heketi-config-secret labeled
service/deploy-heketi created
deployment.apps/deploy-heketi created
Waiting for deploy-heketi pod to start ... OK
Creating cluster ... ID: 2f6eba4d13572a03c0b0af76cd519b79
Allowing file volumes on cluster.
Allowing block volumes on cluster.
Creating node storage-worker-01 ... ID: ed6fea5e1c5f9a7ccb1784e3f5126d4d
Adding device /dev/sdb ... OK
Creating node storage-worker-02 ... ID: 6f1db063cf2853b790156d749dcf1e1c
Adding device /dev/sdb ... OK
Creating node storage-worker-03 ... ID: 4d6ed79b2395317bf305ebb8f20babd2
Adding device /dev/sdb ... OK
heketi topology loaded.
Saving /tmp/heketi-storage.json
secret/heketi-storage-secret created
endpoints/heketi-storage-endpoints created
service/heketi-storage-endpoints created
job.batch/heketi-storage-copy-job created
Error waiting for job 'heketi-storage-copy-job' to complete.
I used the get event command to find out the errors: kubectl get ev --field-selector involvedObject.name=heketi-storage-copy-job-hbvv4
And the output was:
LAST SEEN TYPE REASON OBJECT MESSAGE
<unknown> Normal Scheduled pod/heketi-storage-copy-job-hbvv4 Successfully assigned default/heketi-storage-copy-job-hbvv4 to storage-worker-03
7m49s Warning FailedMount pod/heketi-storage-copy-job-hbvv4 MountVolume.SetUp failed for volume "heketi-storage" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage --scope -- mount -t glusterfs -o auto_unmount,backup-volfile-servers=10.10.2.1:10.10.2.2:10.10.2.3,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/heketi-storage/heketi-storage-copy-job-hbvv4-glusterfs.log,log-level=ERROR 10.10.2.3:heketidbstorage /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage
Output: Running scope as unit: run-r174fd3d54bd2476ab9e056279ea42029.scope
Mount failed. Please check the log file for more details.
, the following error information was pulled from the glusterfs log to help diagnose this issue:
[2020-04-22 06:34:28.998556] E [MSGID: 100026] [glusterfsd.c:2403:glusterfs_process_volfp] 0-: failed to construct the graph
[2020-04-22 06:34:28.998701] E [graph.c:1102:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x537) [0x5556177d7e77] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x180) [0x5556177d1ea0] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7fb786f0eeab] ) 0-graph: invalid argument: graph [Invalid argument]
7m49s Warning FailedMount pod/heketi-storage-copy-job-hbvv4 MountVolume.SetUp failed for volume "heketi-storage" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage --scope -- mount -t glusterfs -o auto_unmount,backup-volfile-servers=10.10.2.1:10.10.2.2:10.10.2.3,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/heketi-storage/heketi-storage-copy-job-hbvv4-glusterfs.log,log-level=ERROR 10.10.2.3:heketidbstorage /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage
Output: Running scope as unit: run-r9787f39108aa4c99a177e556522e9f17.scope
Mount failed. Please check the log file for more details.
, the following error information was pulled from the glusterfs log to help diagnose this issue:
[2020-04-22 06:34:29.663496] E [MSGID: 100026] [glusterfsd.c:2403:glusterfs_process_volfp] 0-: failed to construct the graph
[2020-04-22 06:34:29.663644] E [graph.c:1102:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x537) [0x55dc332fce77] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x180) [0x55dc332f6ea0] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7f056a420eab] ) 0-graph: invalid argument: graph [Invalid argument]
7m48s Warning FailedMount pod/heketi-storage-copy-job-hbvv4 MountVolume.SetUp failed for volume "heketi-storage" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage --scope -- mount -t glusterfs -o auto_unmount,backup-volfile-servers=10.10.2.1:10.10.2.2:10.10.2.3,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/heketi-storage/heketi-storage-copy-job-hbvv4-glusterfs.log,log-level=ERROR 10.10.2.2:heketidbstorage /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage
Output: Running scope as unit: run-r80514c65cb6647f6ac2697b5114f469b.scope
Mount failed. Please check the log file for more details.
, the following error information was pulled from the glusterfs log to help diagnose this issue:
[2020-04-22 06:34:30.779343] E [MSGID: 100026] [glusterfsd.c:2403:glusterfs_process_volfp] 0-: failed to construct the graph
[2020-04-22 06:34:30.779493] E [graph.c:1102:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x537) [0x55f294288e77] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x180) [0x55f294282ea0] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7f070e30beab] ) 0-graph: invalid argument: graph [Invalid argument]
7m45s Warning FailedMount pod/heketi-storage-copy-job-hbvv4 MountVolume.SetUp failed for volume "heketi-storage" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage --scope -- mount -t glusterfs -o auto_unmount,backup-volfile-servers=10.10.2.1:10.10.2.2:10.10.2.3,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/heketi-storage/heketi-storage-copy-job-hbvv4-glusterfs.log,log-level=ERROR 10.10.2.3:heketidbstorage /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage
Output: Running scope as unit: run-r976c7b489f754d9caec29e1da0d1884d.scope
Mount failed. Please check the log file for more details.
, the following error information was pulled from the glusterfs log to help diagnose this issue:
[2020-04-22 06:34:33.011549] E [MSGID: 100026] [glusterfsd.c:2403:glusterfs_process_volfp] 0-: failed to construct the graph
[2020-04-22 06:34:33.011731] E [graph.c:1102:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x537) [0x560cd9c99e77] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x180) [0x560cd9c93ea0] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7f7dcd745eab] ) 0-graph: invalid argument: graph [Invalid argument]
7m41s Warning FailedMount pod/heketi-storage-copy-job-hbvv4 MountVolume.SetUp failed for volume "heketi-storage" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage --scope -- mount -t glusterfs -o auto_unmount,backup-volfile-servers=10.10.2.1:10.10.2.2:10.10.2.3,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/heketi-storage/heketi-storage-copy-job-hbvv4-glusterfs.log,log-level=ERROR 10.10.2.3:heketidbstorage /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage
Output: Running scope as unit: run-rdbd0b7e6802c414b813bd3b188076e4d.scope
Mount failed. Please check the log file for more details.
, the following error information was pulled from the glusterfs log to help diagnose this issue:
[2020-04-22 06:34:37.143898] E [MSGID: 100026] [glusterfsd.c:2403:glusterfs_process_volfp] 0-: failed to construct the graph
[2020-04-22 06:34:37.144040] E [graph.c:1102:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x537) [0x55f51bd1ae77] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x180) [0x55f51bd14ea0] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7febafc8beab] ) 0-graph: invalid argument: graph [Invalid argument]
7m33s Warning FailedMount pod/heketi-storage-copy-job-hbvv4 MountVolume.SetUp failed for volume "heketi-storage" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage --scope -- mount -t glusterfs -o auto_unmount,backup-volfile-servers=10.10.2.1:10.10.2.2:10.10.2.3,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/heketi-storage/heketi-storage-copy-job-hbvv4-glusterfs.log,log-level=ERROR 10.10.2.3:heketidbstorage /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage
Output: Running scope as unit: run-r867c9cb2f25a4f6ba066719ce4ca483d.scope
Mount failed. Please check the log file for more details.
, the following error information was pulled from the glusterfs log to help diagnose this issue:
[2020-04-22 06:34:45.293488] E [MSGID: 100026] [glusterfsd.c:2403:glusterfs_process_volfp] 0-: failed to construct the graph
[2020-04-22 06:34:45.293683] E [graph.c:1102:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x537) [0x55bb171bfe77] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x180) [0x55bb171b9ea0] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7f5d47a95eab] ) 0-graph: invalid argument: graph [Invalid argument]
7m17s Warning FailedMount pod/heketi-storage-copy-job-hbvv4 MountVolume.SetUp failed for volume "heketi-storage" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage --scope -- mount -t glusterfs -o auto_unmount,backup-volfile-servers=10.10.2.1:10.10.2.2:10.10.2.3,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/heketi-storage/heketi-storage-copy-job-hbvv4-glusterfs.log,log-level=ERROR 10.10.2.2:heketidbstorage /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage
Output: Running scope as unit: run-re7d2348c9f334f6eaed9a6216ef332bf.scope
Mount failed. Please check the log file for more details.
, the following error information was pulled from the glusterfs log to help diagnose this issue:
[2020-04-22 06:35:01.489830] E [MSGID: 100026] [glusterfsd.c:2403:glusterfs_process_volfp] 0-: failed to construct the graph
[2020-04-22 06:35:01.489995] E [graph.c:1102:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x537) [0x55b7595fbe77] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x180) [0x55b7595f5ea0] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7f50b6965eab] ) 0-graph: invalid argument: graph [Invalid argument]
6m45s Warning FailedMount pod/heketi-storage-copy-job-hbvv4 MountVolume.SetUp failed for volume "heketi-storage" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage --scope -- mount -t glusterfs -o auto_unmount,backup-volfile-servers=10.10.2.1:10.10.2.2:10.10.2.3,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/heketi-storage/heketi-storage-copy-job-hbvv4-glusterfs.log,log-level=ERROR 10.10.2.2:heketidbstorage /var/lib/kubelet/pods/e449dc88-4642-445b-9994-d183ef3f9b9f/volumes/kubernetes.io~glusterfs/heketi-storage
Output: Running scope as unit: run-r48261be86e40451bb4210c19286f31d8.scope
Mount failed. Please check the log file for more details.
, the following error information was pulled from the glusterfs log to help diagnose this issue:
[2020-04-22 06:35:33.693499] E [MSGID: 100026] [glusterfsd.c:2403:glusterfs_process_volfp] 0-: failed to construct the graph
[2020-04-22 06:35:33.693662] E [graph.c:1102:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x537) [0x55be60e85e77] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x180) [0x55be60e7fea0] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7f20b8b7eeab] ) 0-graph: invalid argument: graph [Invalid argument]
3m30s Warning FailedMount pod/heketi-storage-copy-job-hbvv4 Unable to attach or mount volumes: unmounted volumes=[heketi-storage], unattached volumes=[heketi-storage heketi-storage-secret default-token-t7qfx]: timed out waiting for the condition
76s Warning FailedMount pod/heketi-storage-copy-job-hbvv4 (combined from similar events): Unable to attach or mount volumes: unmounted volumes=[heketi-storage heketi-storage-secret default-token-t7qfx], unattached volumes=[heketi-storage heketi-storage-secret default-token-t7qfx]: timed out waiting for the condition
I'm using Ubuntu Server 18.04
, running kubectl version
yields this:
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1", GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"clean", BuildDate:"2020-04-14T16:45:05Z", GoVersion:"go1.13.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:48:36Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Any help would be appreciated.
I have the same problem I also use Ubuntu 18.04
I moved on, I'm now using rook to install a ceph storage system. You can find a nice guide here
If you are still interested in GlusterFS based solution, please try https://kadalu.io (Project page: https://github.com/kadalu/kadalu)
@aravindavk I will, thanks for bringing this up.