gluster-kubernetes
gluster-kubernetes copied to clipboard
gluster
Heketi failed to create volume: internal error - server connection terminated
When i try to setup a dynamic provisioned volume with heketi-gluster storage class, the volume does not get created but stay in "pending state"
Name: gluster1
Namespace: default
StorageClass: gluster-heketi
Status: Pending
Volume:
Labels: <none>
Annotations: volume.beta.kubernetes.io/storage-class: gluster-heketi
volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/glusterfs
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ProvisioningFailed 35s (x2 over 110s) persistentvolume-controller Failed to provision volume with StorageClass "gluster-heketi": failed to create volume: failed to create volume: <HEAD><TITLE>internal error - server connection terminated</TITLE></HEAD>
<BODY BGCOLOR="white" FGCOLOR="black"><H1>internal error - server connection terminated</H1><HR>
<FONT FACE="Helvetica,Arial"><B>
Description: internal error - server connection terminated</B></FONT>
<HR>
<!-- default "internal error - server connection terminated" response (502) -->
</BODY>
Mounted By: <none>
[kubernetes@ald-kub-01 bin]$ kubectl describe storageclass
Name: gluster-heketi
IsDefaultClass: No
Annotations: <none>
Provisioner: kubernetes.io/glusterfs
Parameters: restauthenabled=false,resturl=http://10.233.79.76:8080
AllowVolumeExpansion: <unset>
MountOptions: <none>
ReclaimPolicy: Delete
VolumeBindingMode: Immediate
Events: <none>
[kubernetes@ald-kub-01 bin]$ export HEKETI_CLI_SERVER=$(kubectl -n glusterfs describe svc heketi |awk '/Endpoints/ {print "http://"$2}')
[kubernetes@ald-kub-01 bin]$ echo $HEKETI_CLI_SERVER
http://10.233.79.76:8080
When using the heketi cli on master server for static volume it seems to work;
[kubernetes@ald-kub-01 bin]$ ./heketi-cli volume create --size=1 --persistent-volume --persistent-volume-endpoint=heketi-storage-endpoints |kubectl create -f -
persistentvolume/glusterfs-733ea120 created
[kubernetes@ald-kub-01 bin]$ kubectl describe pv
Name: glusterfs-733ea120
Labels: <none>
Annotations: <none>
Finalizers: [kubernetes.io/pv-protection]
StorageClass:
Status: Available
Claim:
Reclaim Policy: Retain
Access Modes: RWX
VolumeMode: Filesystem
Capacity: 1Gi
Node Affinity: <none>
Message:
Source:
Type: Glusterfs (a Glusterfs mount on the host that shares a pod's lifetime)
EndpointsName: heketi-storage-endpoints
EndpointsNamespace: <nil>
Path: vol_733ea120c5a2b86e8eb720180ccfc4da
ReadOnly: false
Events: <none>
After some more inspection I suspect that the problem is proxy related due to 10.xxx. is not in the no_proxy list of kubernetes manager
hi @obergeru , i am also seeing similar issue provisioning volume , what are your logs reporting in /var/log/glusterfs?
i am seeing this error where its Samba is not installed!
E [run.c:241:runner_log] (-->/usr/lib64/glusterfs/4.1.7/xlator/mgmt/glusterd.so(+0xe2c9a) [0x7f3d5ac5fc9a] -->/usr/lib64/glusterfs/4.1.7/xlator/mgmt/glusterd.so(+0xe26c3) [0x7f3d5ac5f6c3] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f3d5fdcb0f5] ) 0-management: Failed to execute script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=vol_4eadd2be53ce8fd0a339bcc725cf3932 --first=no --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd
@jarrpa do we need to have Samba installed for the dynamic provisioning? using latest gluster image here.
I have got almost the same error as @obergeru got. I was able to create pv manually with heketi-cli, but when I try to create pvc I see following:
kubectl describe pvc
Name: gluster1
Namespace: default
StorageClass:
Status: Pending
Volume:
Labels: <none>
Annotations: storageClassName: gluster-heketi
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Mounted By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal FailedBinding 3m49s (x62 over 18m) persistentvolume-controller no persistent volumes available for this claim and no storage class is set
@maurya-m Samba is not required nor recommended for provisioning gluster volumes with heketi. That error seems to be a problem running a hook script in gluster. Is this a standalone gluster setup you built or is it a containerized gluster where you see this logging?
@poligraph This does not appear to be the same error. Your error has the text "no storage class is set" which indicates your PVC was not set up correctly for dynamic provisioning. Please review your PVC yaml your cluster needs either a default storage class or the storage class to be specificd in the pvc.
