linstor-csi
linstor-csi copied to clipboard
ReadWriteMany accessMode
Hi,
I'm trying to create a StatefulSet
with a ReadWriteMany
accessMode.
It doesn't work. Is it possible to do ReadWriteMany
access mode with Linstor and Lisntor CSI ?
Events on test-0
:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling <unknown> default-scheduler persistentvolumeclaim "data" not found
Normal Scheduled <unknown> default-scheduler Successfully assigned default/test-0 to node2
Normal SuccessfulAttachVolume 8m18s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196"
Warning FailedMount 7m31s (x3 over 8m3s) kubelet, node2 MountVolume.SetUp failed for volume "pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196" : kubernetes.io/csi: mounter.SetupAt failed: rpc error: code = Internal desc = NodePublishVolume failed for pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196: mounting volume failed: invalid parameter: resourcegroup does not belong to paramKey values
Warning FailedMount 3m58s (x2 over 6m16s) kubelet, node2 Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[default-token-m9cnr data]: timed out waiting for the condition
Warning FailedMount 111s (x8 over 8m3s) kubelet, node2 MountVolume.SetUp failed for volume "pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196" : kubernetes.io/csi: mounter.SetupAt failed: rpc error: code = Internal desc = NodePublishVolume failed for pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196: mounting volume failed: invalid parameter: drbdoptions/net/allow-two-primaries does not belong to paramKey values
Warning FailedMount 104s kubelet, node2 Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data default-token-m9cnr]: timed out waiting for the condition
kubectl logs linstor-csi-node-pvdjb -n linstor -c linstor-csi-plugin
:
time="2020-04-30T06:13:33Z" level=debug msg="method called" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:277" linstorCSIComponent=driver method=/csi.v1.Node/NodeGetCapabilities nodeID=node2 provisioner=linstor.csi.linbit.com req= resp="capabilities:<rpc:<type:GET_VOLUME_STATS > > " version=v0.7.4-9-g7383849
time="2020-04-30T06:13:33Z" level=debug msg="looking up resource by CSI volume id" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:277" csiVolumeID=pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 linstorCSIComponent=client
time="2020-04-30T06:13:33Z" level=info msg="curl -X 'GET' -H 'Accept: application/json' 'https://linstor-controller:3371/v1/resource-definitions/pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196'"
time="2020-04-30T06:13:33Z" level=debug msg="converted resource definition to volume" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:277" linstorCSIComponent=client resourceDefinition="{Name:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 ExternalName:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 Props:map[Aux/csi-volume-annotations:{\"name\":\"pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196\",\"id\":\"pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2020-04-30T06:03:00.12667645Z\",\"sizeBytes\":1073741824,\"readonly\":false,\"parameters\":{\"DrbdOptions/Net/allow-two-primaries\":\"yes\",\"resourceGroup\":\"linstor_group\",\"storagePool\":\"pool_linstor\"},\"snapshots\":[]} DrbdOptions/Net/allow-two-primaries:yes DrbdOptions/Resource/quorum:off DrbdPrimarySetOn:NODE1 StorPoolName:pool_linstor] Flags:[] LayerData:[{Type:DRBD Data:0xc0003d2e10} {Type:STORAGE Data:<nil>}] Uuid:d4b77817-bf44-405a-93c0-c40035fca705}" volume="&{Name:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 ID:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 CreatedBy:linstor.csi.linbit.com CreationTime:2020-04-30 06:03:00.12667645 +0000 UTC SizeBytes:1073741824 Readonly:false Parameters:map[DrbdOptions/Net/allow-two-primaries:yes resourceGroup:linstor_group storagePool:pool_linstor] Snapshots:[]}"
time="2020-04-30T06:13:33Z" level=debug msg="getting assignment info" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:277" linstorCSIComponent=client targetNode=node2 volume="&{Name:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 ID:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 CreatedBy:linstor.csi.linbit.com CreationTime:2020-04-30 06:03:00.12667645 +0000 UTC SizeBytes:1073741824 Readonly:false Parameters:map[DrbdOptions/Net/allow-two-primaries:yes resourceGroup:linstor_group storagePool:pool_linstor] Snapshots:[]}"
time="2020-04-30T06:13:33Z" level=info msg="curl -X 'GET' -H 'Accept: application/json' 'https://linstor-controller:3371/v1/resource-definitions/pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196/resources/node2/volumes/0'"
time="2020-04-30T06:13:33Z" level=debug msg="found assignment info" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:277" linstorCSIComponent=client volumeAssignment="&{Vol:0xc000208800 Node:node2 Path:/dev/drbd1000}"
time="2020-04-30T06:13:33Z" level=error msg="method failed" func="github.com/sirupsen/logrus.(*Entry).Error" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:297" error="rpc error: code = Internal desc = NodePublishVolume failed for pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196: mounting volume failed: invalid parameter: drbdoptions/net/allow-two-primaries does not belong to paramKey values" linstorCSIComponent=driver method=/csi.v1.Node/NodePublishVolume nodeID=node2 provisioner=linstor.csi.linbit.com req="volume_id:\"pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196\" target_path:\"/var/lib/kubelet/pods/eecc1e3b-df42-4012-a2d9-24fadf59ef18/volumes/kubernetes.io~csi/pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196/mount\" volume_capability:<mount:<fs_type:\"ext4\" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > > volume_context:<key:\"csi.storage.k8s.io/ephemeral\" value:\"false\" > volume_context:<key:\"csi.storage.k8s.io/pod.name\" value:\"test-0\" > volume_context:<key:\"csi.storage.k8s.io/pod.namespace\" value:\"default\" > volume_context:<key:\"csi.storage.k8s.io/pod.uid\" value:\"eecc1e3b-df42-4012-a2d9-24fadf59ef18\" > volume_context:<key:\"csi.storage.k8s.io/serviceAccount.name\" value:\"default\" > volume_context:<key:\"storage.kubernetes.io/csiProvisionerIdentity\" value:\"1588184499057-8081-linstor.csi.linbit.com\" > " resp="<nil>" version=v0.7.4-9-g7383849
linstor controller
$ linstor storage-pool list
+----------------------------------------------------------------------------------------------------------------------------------+
| StoragePool | Node | Driver | PoolName | FreeCapacity | TotalCapacity | CanSnapshots | State |
|==================================================================================================================================|
| DfltDisklessStorPool | node1 | DISKLESS | | | | False | Ok |
| DfltDisklessStorPool | node2 | DISKLESS | | | | False | Ok |
| pool_linstor | node1 | LVM | vg_linstor | 334.34 GiB | 335.35 GiB | False | Ok |
| pool_linstor | node2 | LVM | vg_linstor | 335.35 GiB | 335.35 GiB | False | Ok |
+----------------------------------------------------------------------------------------------------------------------------------+
$
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Node | Resource | StoragePool | VolNr | MinorNr | DeviceName | Allocated | InUse | State |
|=================================================================================================================================================================|
| node1 | pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 | pool_linstor | 0 | 1000 | /dev/drbd1000 | 1.00 GiB | Unused | UpToDate |
| node2 | pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 | DfltDisklessStorPool | 0 | 1000 | /dev/drbd1000 | | Unused | Diskless |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
$ linstor resource-group list
+------------------------------------------------------------------------+
| ResourceGroup | SelectFilter | VlmNrs | Description |
|========================================================================|
| DfltRscGrp | PlaceCount: 2 | | |
|------------------------------------------------------------------------|
| linstor_group | PlaceCount: 1 | 0 | |
| | StoragePool: pool_linstor | | |
| | LayerStack: ['DRBD', 'STORAGE'] | | |
+------------------------------------------------------------------------+
$ linstor resource-group list-properties linstor_group
+----------------------------------------------------+
| Key | Value |
|====================================================|
| DrbdOptions/Net/allow-two-primaries | yes |
| StorPoolName | pool_linstor |
+----------------------------------------------------+
drbdadm status
on node2
pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 role:Secondary
disk:Diskless
node1 role:Secondary
peer-disk:UpToDate
drbdsetup events
on node2
exists resource name:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 role:Secondary suspended:no
exists connection name:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 peer-node-id:0 conn-name:node1 connection:Connected role:Secondary
exists device name:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 volume:0 minor:1000 disk:Diskless client:yes quorum:yes
exists peer-device name:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 peer-node-id:0 conn-name:node1 volume:0 replication:Established peer-disk:UpToDate peer-client:no resync-suspended:no
exists -
Definition
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
# The name used to identify this StorageClass.
name: linstor-basic-storage-class
# The name used to match this StorageClass with a provisioner.
# linstor.csi.linbit.com is the name that the LINSTOR CSI plugin uses to identify itself
provisioner: linstor.csi.linbit.com
parameters:
# LINSTOR will provision volumes from the drbdpool storage pool configured
# On the satellite nodes in the LINSTOR cluster specified in the plugin's deployment
storagePool: "pool_linstor"
resourceGroup: "linstor_group"
DrbdOptions/Net/allow-two-primaries: "yes"
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: test
namespace: default
spec:
replicas: 2
selector:
matchLabels:
app: test
serviceName: test
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: test
spec:
securityContext:
fsGroup: 1001
runAsUser: 1001
containers:
- name: debug
image: alpine:3.11
command: ["/bin/sh"]
stdin: true
tty: true
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: data
readOnly: false
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data
namespace: default
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
storageClassName: linstor-basic-storage-class
---
This is being worked on. By default you get a mounted directory with a file system on a DRBD device. The DRBD device/the resource (per default) is not allowed to be dual primary, and even if so, having multiple writers on the file system would break anyways.
The only thing supported right now is raw block mode, with setting DRBD two-primaries via the storage class and having them dual primary while live migration (e.g., kubevirt).
OK. Thanks for the reply. I gone further with :
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
# The name used to identify this StorageClass.
name: linstor-basic-storage-class
# The name used to match this StorageClass with a provisioner.
# linstor.csi.linbit.com is the name that the LINSTOR CSI plugin uses to identify itself
provisioner: linstor.csi.linbit.com
parameters:
# LINSTOR will provision volumes from the drbdpool storage pool configured
# On the satellite nodes in the LINSTOR cluster specified in the plugin's deployment
storagePool: "pool_linstor"
DrbdOptions/Net/allow-two-primaries: "yes"
placementCount: "2"
placementPolicy: "AutoPlace"
Here my volumes are OK on the two nodes with the resource created correctly (no Diskless). But still in error.
Can I help somehow on this ?
Exactly, but now you mount the FS on two nodes, which really really is not a good idea. Contributions are always welcome, but AFAIK there is a team working on it, so just waiting might be the best for now. @alexzhc , how is the progress on that?
OK. I was able to do ReadWriteMany access with Rook and Ceph but performances are better with Linstor. My goal is to create load balancing on web app with access to the same filesystem to store and read files.
Hi!
I have a very similar use-case as @jbanety commented above: I have many applications, which I want to horizontally scale and need to have access to a shared FS.
@rck: You mentioned work was being done on it. Is it open source? If so where can I find more information or the ongoing development?
I don't know, I'm not working on this part, @alexzhc is still the one that could give updates on it.
Hi, Would also like to combine Linstor's performance and ReadWriteMany access for our usecase. An update to have a rough idea of when / if this might happen would be very helpful @alexzhc , thanks.
Integrating an NFS export by using a NFS-Ganesha POD on top of a drbd RWO volume is not hard as demonstrated by Rook-NFS. However, by doing so, the NFS-Ganesha POD with the drbd RWO volume needs to failover in a reasonable RTO in case of node failure. However, besides 5-minutes k8s node timeout, currently in CSI, RWO volume attachment failover has a hard-coded timeout of 6 minutes. Such a long failover time will definitely cause NFS mount to go stale.
We need to solve the RWO failover RTO issue before implementing NFS export.
Hi, take a look at nfs-server-provisioner project, we're using it together with linstor for having ReadWriteMany volumes.
Failover is ensured by kube-fencing.
Hi @kvaps, it sounds interesting. Are NFS performances good enough for a heavy loaded web app? Do you have a working example with linstor, NFS provisionnel and kube-fencing?
Are NFS performances good enough for a heavy loaded web app?
Performance is fine, you can see my benchmarks for comparison: https://gist.github.com/kvaps/8c2831ca15cf161a11e62cc4276793f7
Do you have a working example with linstor, NFS provisionnel and kube-fencing
For the nfs-server-provisioner, we're using the helm chart: https://github.com/kvaps/nfs-server-provisioner-chart
here is example values:
persistence:
enabled: true
size: 50Gi
storageClass: linstor-1
storageClass:
name: example-nfs
After deployment, you can request new volume using example-nfs
storageclass
Fencing is very specific to your environment, here is an example for HP ilo https://raw.githubusercontent.com/kvaps/kube-fencing/master/deploy/examples/hp-ilo.yaml
Meanwhile I would advise to avoid using ReadWriteMany in case if you don't really needed it, since it is adding extra abstraction layers, which might made whole system less stable.
Consider using S3 (minio) in your application instead
Meanwhile I would advise to avoid using ReadWriteMany in case if you don't really needed it, since it is adding extra abstraction layers, which might made whole system less stable.
Consider using S3 (minio) in your application instead
Yes. I wanted to have a deployment with two pods on two different nodes to serve the same Prestashop app for load balancing. It must sharing a file system for image uploads,... I can't use S3 by design. As a workaround, I created a StatefulSet with only 1 replica to fail over on the other node but no load balancing.
Well we're using same approach to host Nextcloud itself, but the user data is stored on another server using S3 backend
@kvaps We have tried using https://github.com/kubernetes-sigs/nfs-ganesha-server-and-external-provisioner per your suggestion with a pireaus StorageClass, however when the NFS pod is rescheduled on to another node due to node failure, the mount on the new node via the NFS service just hangs.
[pid 1729737] openat(AT_FDCWD, "/etc/protocols", O_RDONLY|O_CLOEXEC) = 3
[pid 1729737] lseek(3, 0, SEEK_CUR) = 0
[pid 1729737] fstat(3, {st_mode=S_IFREG|0644, st_size=2932, ...}) = 0
[pid 1729737] read(3, "# Internet (IP) protocols\n#\n# Up"..., 3072) = 2932
[pid 1729737] close(3) = 0
[pid 1729737] openat(AT_FDCWD, "/etc/netconfig", O_RDONLY) = 3
[pid 1729737] fstat(3, {st_mode=S_IFREG|0644, st_size=767, ...}) = 0
[pid 1729737] read(3, "#\n# The network configuration fi"..., 1024) = 767
[pid 1729737] close(3) = 0
[pid 1729737] openat(AT_FDCWD, "/etc/protocols", O_RDONLY|O_CLOEXEC) = 3
[pid 1729737] lseek(3, 0, SEEK_CUR) = 0
[pid 1729737] fstat(3, {st_mode=S_IFREG|0644, st_size=2932, ...}) = 0
[pid 1729737] read(3, "# Internet (IP) protocols\n#\n# Up"..., 3072) = 2932
[pid 1729737] close(3) = 0
[pid 1729737] openat(AT_FDCWD, "/etc/netconfig", O_RDONLY) = 3
[pid 1729737] fstat(3, {st_mode=S_IFREG|0644, st_size=767, ...}) = 0
[pid 1729737] read(3, "#\n# The network configuration fi"..., 1024) = 767
[pid 1729737] close(3) = 0
[pid 1729737] mount("10.71.214.117:/export/pvc-aad0a19d-8cfa-4344-b58c-59f95acfccf7", "/nfs", "nfs", 0, "addr=10.71.214.117,vers=3,proto="...
We've tried nfsver 3 & 4 and are having the same issue.
Weirdly you can still connect via NFS on the same service if you change the nfsver parameter, but the original nfsver setting does not work.
Have you experienced this issue, do you have any advice which may help?
Sounds like network problem May I get know, what CNI and which mode of kube-proxy you're using?
@kvaps thanks for the reply, I agree. Currently trying to debug the issue with tcpdump/conntrack & cilium. The CNI is cilium, unfortunately tcpdump on the service IP doesn't show anything as the routing is done in BPF.
I can see the packets getting through to the correct endpoint (10.0.0.252
below), but after switching to UDP the packets stop (they don't seem to be dropped).
15:02:25.202882 IP (tos 0x0, ttl 64, id 50089, offset 0, flags [DF], proto TCP (6), length 60)
10.0.4.242.34278 > 10.0.0.252.111: Flags [S], cksum 0x1a1c (incorrect -> 0xb9bc), seq 800816836, win 64860, options [mss 1410,sackOK,TS val 1768478956 ecr 0,nop,wscale 8], length 0
0x0000: 4500 003c c3a9 4000 4006 5d25 0a00 04f2 E..<..@.@.]%....
0x0010: 0a00 00fc 85e6 006f 2fbb 7ec4 0000 0000 .......o/.~.....
0x0020: a002 fd5c 1a1c 0000 0204 0582 0402 080a ...\............
0x0030: 6968 d8ec 0000 0000 0103 0308 ih..........
15:02:25.203303 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
10.0.0.252.111 > 10.0.4.242.34278: Flags [S.], cksum 0xc601 (correct), seq 2873377936, ack 800816837, win 64308, options [mss 1410,sackOK,TS val 1826134308 ecr 1768478956,nop,wscale 8], length 0
0x0000: 4500 003c 0000 4000 4006 20cf 0a00 00fc E..<..@.@.......
0x0010: 0a00 04f2 006f 85e6 ab44 4490 2fbb 7ec5 .....o...DD./.~.
0x0020: a012 fb34 c601 0000 0204 0582 0402 080a ...4............
0x0030: 6cd8 9924 6968 d8ec 0103 0308 l..$ih......
15:02:25.203446 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
10.0.0.252.111 > 10.0.4.242.34278: Flags [S.], cksum 0xc601 (correct), seq 2873377936, ack 800816837, win 64308, options [mss 1410,sackOK,TS val 1826134308 ecr 1768478956,nop,wscale 8], length 0
0x0000: 4500 003c 0000 4000 3f06 21cf 0a00 00fc E..<..@.?.!.....
0x0010: 0a00 04f2 006f 85e6 ab44 4490 2fbb 7ec5 .....o...DD./.~.
0x0020: a012 fb34 c601 0000 0204 0582 0402 080a ...4............
0x0030: 6cd8 9924 6968 d8ec 0103 0308 l..$ih......
15:02:25.203449 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
10.0.0.252.111 > 10.0.4.242.34278: Flags [S.], cksum 0xc601 (correct), seq 2873377936, ack 800816837, win 64308, options [mss 1410,sackOK,TS val 1826134308 ecr 1768478956,nop,wscale 8], length 0
0x0000: 4500 003c 0000 4000 3f06 21cf 0a00 00fc E..<..@.?.!.....
0x0010: 0a00 04f2 006f 85e6 ab44 4490 2fbb 7ec5 .....o...DD./.~.
0x0020: a012 fb34 c601 0000 0204 0582 0402 080a ...4............
0x0030: 6cd8 9924 6968 d8ec 0103 0308 l..$ih......
15:02:25.203638 IP (tos 0x0, ttl 64, id 50090, offset 0, flags [DF], proto TCP (6), length 52)
10.0.4.242.34278 > 10.0.0.252.111: Flags [.], cksum 0x1a14 (incorrect -> 0xeed2), seq 1, ack 1, win 254, options [nop,nop,TS val 1768478957 ecr 1826134308], length 0
0x0000: 4500 0034 c3aa 4000 4006 5d2c 0a00 04f2 E..4..@.@.],....
0x0010: 0a00 00fc 85e6 006f 2fbb 7ec5 ab44 4491 .......o/.~..DD.
0x0020: 8010 00fe 1a14 0000 0101 080a 6968 d8ed ............ih..
0x0030: 6cd8 9924 l..$
15:02:25.231179 IP (tos 0x0, ttl 64, id 50091, offset 0, flags [DF], proto TCP (6), length 112)
10.0.4.242.34278 > 10.0.0.252.111: Flags [P.], cksum 0x1a50 (incorrect -> 0x4c09), seq 1:61, ack 1, win 254, options [nop,nop,TS val 1768478985 ecr 1826134308], length 60
0x0000: 4500 0070 c3ab 4000 4006 5cef 0a00 04f2 E..p..@.@.\.....
0x0010: 0a00 00fc 85e6 006f 2fbb 7ec5 ab44 4491 .......o/.~..DD.
0x0020: 8018 00fe 1a50 0000 0101 080a 6968 d909 .....P......ih..
0x0030: 6cd8 9924 8000 0038 d2ad 422d 0000 0000 l..$...8..B-....
0x0040: 0000 0002 0001 86a0 0000 0002 0000 0003 ................
0x0050: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0060: 0001 86a3 0000 0003 0000 0006 0000 0000 ................
15:02:25.231567 IP (tos 0x0, ttl 64, id 45867, offset 0, flags [DF], proto TCP (6), length 52)
10.0.0.252.111 > 10.0.4.242.34278: Flags [.], cksum 0xee61 (correct), seq 1, ack 61, win 251, options [nop,nop,TS val 1826134336 ecr 1768478985], length 0
0x0000: 4500 0034 b32b 4000 4006 6dab 0a00 00fc E..4.+@[email protected].....
0x0010: 0a00 04f2 006f 85e6 ab44 4491 2fbb 7f01 .....o...DD./...
0x0020: 8010 00fb ee61 0000 0101 080a 6cd8 9940 .....a......l..@
0x0030: 6968 d909 ih..
15:02:25.231667 IP (tos 0x0, ttl 63, id 45867, offset 0, flags [DF], proto TCP (6), length 52)
10.0.0.252.111 > 10.0.4.242.34278: Flags [.], cksum 0xee61 (correct), seq 1, ack 61, win 251, options [nop,nop,TS val 1826134336 ecr 1768478985], length 0
0x0000: 4500 0034 b32b 4000 3f06 6eab 0a00 00fc E..4.+@.?.n.....
0x0010: 0a00 04f2 006f 85e6 ab44 4491 2fbb 7f01 .....o...DD./...
0x0020: 8010 00fb ee61 0000 0101 080a 6cd8 9940 .....a......l..@
0x0030: 6968 d909 ih..
15:02:25.231567 IP (tos 0x0, ttl 64, id 45868, offset 0, flags [DF], proto TCP (6), length 84)
10.0.0.252.111 > 10.0.4.242.34278: Flags [P.], cksum 0x5140 (correct), seq 1:33, ack 61, win 251, options [nop,nop,TS val 1826134336 ecr 1768478985], length 32
0x0000: 4500 0054 b32c 4000 4006 6d8a 0a00 00fc E..T.,@[email protected].....
0x0010: 0a00 04f2 006f 85e6 ab44 4491 2fbb 7f01 .....o...DD./...
0x0020: 8018 00fb 5140 0000 0101 080a 6cd8 9940 [email protected]..@
0x0030: 6968 d909 8000 001c d2ad 422d 0000 0001 ih........B-....
0x0040: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0050: 0000 0801 ....
15:02:25.231675 IP (tos 0x0, ttl 63, id 45868, offset 0, flags [DF], proto TCP (6), length 84)
10.0.0.252.111 > 10.0.4.242.34278: Flags [P.], cksum 0x5140 (correct), seq 1:33, ack 61, win 251, options [nop,nop,TS val 1826134336 ecr 1768478985], length 32
0x0000: 4500 0054 b32c 4000 3f06 6e8a 0a00 00fc E..T.,@.?.n.....
0x0010: 0a00 04f2 006f 85e6 ab44 4491 2fbb 7f01 .....o...DD./...
0x0020: 8018 00fb 5140 0000 0101 080a 6cd8 9940 [email protected]..@
0x0030: 6968 d909 8000 001c d2ad 422d 0000 0001 ih........B-....
0x0040: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0050: 0000 0801 ....
15:02:25.231671 IP (tos 0x0, ttl 63, id 45867, offset 0, flags [DF], proto TCP (6), length 52)
10.0.0.252.111 > 10.0.4.242.34278: Flags [.], cksum 0xee61 (correct), seq 1, ack 61, win 251, options [nop,nop,TS val 1826134336 ecr 1768478985], length 0
0x0000: 4500 0034 b32b 4000 3f06 6eab 0a00 00fc E..4.+@.?.n.....
0x0010: 0a00 04f2 006f 85e6 ab44 4491 2fbb 7f01 .....o...DD./...
0x0020: 8010 00fb ee61 0000 0101 080a 6cd8 9940 .....a......l..@
0x0030: 6968 d909 ih..
15:02:25.231676 IP (tos 0x0, ttl 63, id 45868, offset 0, flags [DF], proto TCP (6), length 84)
10.0.0.252.111 > 10.0.4.242.34278: Flags [P.], cksum 0x5140 (correct), seq 1:33, ack 61, win 251, options [nop,nop,TS val 1826134336 ecr 1768478985], length 32
0x0000: 4500 0054 b32c 4000 3f06 6e8a 0a00 00fc E..T.,@.?.n.....
0x0010: 0a00 04f2 006f 85e6 ab44 4491 2fbb 7f01 .....o...DD./...
0x0020: 8018 00fb 5140 0000 0101 080a 6cd8 9940 [email protected]..@
0x0030: 6968 d909 8000 001c d2ad 422d 0000 0001 ih........B-....
0x0040: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0050: 0000 0801 ....
15:02:25.231857 IP (tos 0x0, ttl 64, id 50092, offset 0, flags [DF], proto TCP (6), length 52)
10.0.4.242.34278 > 10.0.0.252.111: Flags [.], cksum 0x1a14 (incorrect -> 0xee3e), seq 61, ack 33, win 254, options [nop,nop,TS val 1768478985 ecr 1826134336], length 0
0x0000: 4500 0034 c3ac 4000 4006 5d2a 0a00 04f2 E..4..@.@.]*....
0x0010: 0a00 00fc 85e6 006f 2fbb 7f01 ab44 44b1 .......o/....DD.
0x0020: 8010 00fe 1a14 0000 0101 080a 6968 d909 ............ih..
0x0030: 6cd8 9940 l..@
15:02:25.232044 IP (tos 0x0, ttl 64, id 50093, offset 0, flags [DF], proto TCP (6), length 52)
10.0.4.242.34278 > 10.0.0.252.111: Flags [F.], cksum 0x1a14 (incorrect -> 0xee3c), seq 61, ack 33, win 254, options [nop,nop,TS val 1768478986 ecr 1826134336], length 0
0x0000: 4500 0034 c3ad 4000 4006 5d29 0a00 04f2 E..4..@.@.])....
0x0010: 0a00 00fc 85e6 006f 2fbb 7f01 ab44 44b1 .......o/....DD.
0x0020: 8011 00fe 1a14 0000 0101 080a 6968 d90a ............ih..
0x0030: 6cd8 9940 l..@
15:02:25.232293 IP (tos 0x0, ttl 64, id 36895, offset 0, flags [DF], proto TCP (6), length 60)
10.0.4.242.49762 > 10.0.0.252.2049: Flags [S], cksum 0x1a1c (incorrect -> 0x081a), seq 3636937518, win 64860, options [mss 1410,sackOK,TS val 1768478986 ecr 0,nop,wscale 8], length 0
0x0000: 4500 003c 901f 4000 4006 90af 0a00 04f2 E..<..@.@.......
0x0010: 0a00 00fc c262 0801 d8c7 432e 0000 0000 .....b....C.....
0x0020: a002 fd5c 1a1c 0000 0204 0582 0402 080a ...\............
0x0030: 6968 d90a 0000 0000 0103 0308 ih..........
15:02:25.232407 IP (tos 0x0, ttl 64, id 45869, offset 0, flags [DF], proto TCP (6), length 52)
10.0.0.252.111 > 10.0.4.242.34278: Flags [F.], cksum 0xee3d (correct), seq 33, ack 62, win 251, options [nop,nop,TS val 1826134337 ecr 1768478986], length 0
0x0000: 4500 0034 b32d 4000 4006 6da9 0a00 00fc E..4.-@[email protected].....
0x0010: 0a00 04f2 006f 85e6 ab44 44b1 2fbb 7f02 .....o...DD./...
0x0020: 8011 00fb ee3d 0000 0101 080a 6cd8 9941 .....=......l..A
0x0030: 6968 d90a ih..
15:02:25.232461 IP (tos 0x0, ttl 63, id 45869, offset 0, flags [DF], proto TCP (6), length 52)
10.0.0.252.111 > 10.0.4.242.34278: Flags [F.], cksum 0xee3d (correct), seq 33, ack 62, win 251, options [nop,nop,TS val 1826134337 ecr 1768478986], length 0
0x0000: 4500 0034 b32d 4000 3f06 6ea9 0a00 00fc E..4.-@.?.n.....
0x0010: 0a00 04f2 006f 85e6 ab44 44b1 2fbb 7f02 .....o...DD./...
0x0020: 8011 00fb ee3d 0000 0101 080a 6cd8 9941 .....=......l..A
0x0030: 6968 d90a ih..
15:02:25.232463 IP (tos 0x0, ttl 63, id 45869, offset 0, flags [DF], proto TCP (6), length 52)
10.0.0.252.111 > 10.0.4.242.34278: Flags [F.], cksum 0xee3d (correct), seq 33, ack 62, win 251, options [nop,nop,TS val 1826134337 ecr 1768478986], length 0
0x0000: 4500 0034 b32d 4000 3f06 6ea9 0a00 00fc E..4.-@.?.n.....
0x0010: 0a00 04f2 006f 85e6 ab44 44b1 2fbb 7f02 .....o...DD./...
0x0020: 8011 00fb ee3d 0000 0101 080a 6cd8 9941 .....=......l..A
0x0030: 6968 d90a ih..
15:02:25.232581 IP (tos 0x0, ttl 64, id 50094, offset 0, flags [DF], proto TCP (6), length 52)
10.0.4.242.34278 > 10.0.0.252.111: Flags [.], cksum 0x1a14 (incorrect -> 0xee3a), seq 62, ack 34, win 254, options [nop,nop,TS val 1768478986 ecr 1826134337], length 0
0x0000: 4500 0034 c3ae 4000 4006 5d28 0a00 04f2 E..4..@.@.](....
0x0010: 0a00 00fc 85e6 006f 2fbb 7f02 ab44 44b2 .......o/....DD.
0x0020: 8010 00fe 1a14 0000 0101 080a 6968 d90a ............ih..
0x0030: 6cd8 9941 l..A
15:02:25.232637 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
10.0.0.252.2049 > 10.0.4.242.49762: Flags [S.], cksum 0xc65f (correct), seq 312290074, ack 3636937519, win 64308, options [mss 1410,sackOK,TS val 1826134337 ecr 1768478986,nop,wscale 8], length 0
0x0000: 4500 003c 0000 4000 4006 20cf 0a00 00fc E..<..@.@.......
0x0010: 0a00 04f2 0801 c262 129d 2b1a d8c7 432f .......b..+...C/
0x0020: a012 fb34 c65f 0000 0204 0582 0402 080a ...4._..........
0x0030: 6cd8 9941 6968 d90a 0103 0308 l..Aih......
15:02:25.232677 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
10.0.0.252.2049 > 10.0.4.242.49762: Flags [S.], cksum 0xc65f (correct), seq 312290074, ack 3636937519, win 64308, options [mss 1410,sackOK,TS val 1826134337 ecr 1768478986,nop,wscale 8], length 0
0x0000: 4500 003c 0000 4000 3f06 21cf 0a00 00fc E..<..@.?.!.....
0x0010: 0a00 04f2 0801 c262 129d 2b1a d8c7 432f .......b..+...C/
0x0020: a012 fb34 c65f 0000 0204 0582 0402 080a ...4._..........
0x0030: 6cd8 9941 6968 d90a 0103 0308 l..Aih......
15:02:25.232681 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
10.0.0.252.2049 > 10.0.4.242.49762: Flags [S.], cksum 0xc65f (correct), seq 312290074, ack 3636937519, win 64308, options [mss 1410,sackOK,TS val 1826134337 ecr 1768478986,nop,wscale 8], length 0
0x0000: 4500 003c 0000 4000 3f06 21cf 0a00 00fc E..<..@.?.!.....
0x0010: 0a00 04f2 0801 c262 129d 2b1a d8c7 432f .......b..+...C/
0x0020: a012 fb34 c65f 0000 0204 0582 0402 080a ...4._..........
0x0030: 6cd8 9941 6968 d90a 0103 0308 l..Aih......
15:02:25.232814 IP (tos 0x0, ttl 64, id 36896, offset 0, flags [DF], proto TCP (6), length 52)
10.0.4.242.49762 > 10.0.0.252.2049: Flags [.], cksum 0x1a14 (incorrect -> 0xef31), seq 1, ack 1, win 254, options [nop,nop,TS val 1768478986 ecr 1826134337], length 0
0x0000: 4500 0034 9020 4000 4006 90b6 0a00 04f2 E..4..@.@.......
0x0010: 0a00 00fc c262 0801 d8c7 432f 129d 2b1b .....b....C/..+.
0x0020: 8010 00fe 1a14 0000 0101 080a 6968 d90a ............ih..
0x0030: 6cd8 9941 l..A
15:02:25.233022 IP (tos 0x0, ttl 64, id 36897, offset 0, flags [DF], proto TCP (6), length 96)
10.0.4.242.49762 > 10.0.0.252.2049: Flags [P.], cksum 0x1a40 (incorrect -> 0xcbcd), seq 1:45, ack 1, win 254, options [nop,nop,TS val 1768478987 ecr 1826134337], length 44: NFS request xid 3534571951 40 null
0x0000: 4500 0060 9021 4000 4006 9089 0a00 04f2 E..`.!@.@.......
0x0010: 0a00 00fc c262 0801 d8c7 432f 129d 2b1b .....b....C/..+.
0x0020: 8018 00fe 1a40 0000 0101 080a 6968 d90b [email protected]..
0x0030: 6cd8 9941 8000 0028 d2ad 49af 0000 0000 l..A...(..I.....
0x0040: 0000 0002 0001 86a3 0000 0003 0000 0000 ................
0x0050: 0000 0000 0000 0000 0000 0000 0000 0000 ................
15:02:25.233357 IP (tos 0x0, ttl 64, id 41130, offset 0, flags [DF], proto TCP (6), length 52)
10.0.0.252.2049 > 10.0.4.242.49762: Flags [.], cksum 0xef05 (correct), seq 1, ack 45, win 252, options [nop,nop,TS val 1826134338 ecr 1768478987], length 0
0x0000: 4500 0034 a0aa 4000 4006 802c 0a00 00fc E..4..@.@..,....
0x0010: 0a00 04f2 0801 c262 129d 2b1b d8c7 435b .......b..+...C[
0x0020: 8010 00fc ef05 0000 0101 080a 6cd8 9942 ............l..B
0x0030: 6968 d90b ih..
15:02:25.233407 IP (tos 0x0, ttl 63, id 41130, offset 0, flags [DF], proto TCP (6), length 52)
10.0.0.252.2049 > 10.0.4.242.49762: Flags [.], cksum 0xef05 (correct), seq 1, ack 45, win 252, options [nop,nop,TS val 1826134338 ecr 1768478987], length 0
0x0000: 4500 0034 a0aa 4000 3f06 812c 0a00 00fc E..4..@.?..,....
0x0010: 0a00 04f2 0801 c262 129d 2b1b d8c7 435b .......b..+...C[
0x0020: 8010 00fc ef05 0000 0101 080a 6cd8 9942 ............l..B
0x0030: 6968 d90b ih..
15:02:25.233411 IP (tos 0x0, ttl 63, id 41130, offset 0, flags [DF], proto TCP (6), length 52)
10.0.0.252.2049 > 10.0.4.242.49762: Flags [.], cksum 0xef05 (correct), seq 1, ack 45, win 252, options [nop,nop,TS val 1826134338 ecr 1768478987], length 0
0x0000: 4500 0034 a0aa 4000 3f06 812c 0a00 00fc E..4..@.?..,....
0x0010: 0a00 04f2 0801 c262 129d 2b1b d8c7 435b .......b..+...C[
0x0020: 8010 00fc ef05 0000 0101 080a 6cd8 9942 ............l..B
0x0030: 6968 d90b ih..
15:02:25.233619 IP (tos 0x0, ttl 64, id 41131, offset 0, flags [DF], proto TCP (6), length 80)
10.0.0.252.2049 > 10.0.4.242.49762: Flags [P.], cksum 0x526b (correct), seq 1:29, ack 45, win 252, options [nop,nop,TS val 1826134338 ecr 1768478987], length 28: NFS reply xid 3534571951 reply ok 24 null
0x0000: 4500 0050 a0ab 4000 4006 800f 0a00 00fc E..P..@.@.......
0x0010: 0a00 04f2 0801 c262 129d 2b1b d8c7 435b .......b..+...C[
0x0020: 8018 00fc 526b 0000 0101 080a 6cd8 9942 ....Rk......l..B
0x0030: 6968 d90b 8000 0018 d2ad 49af 0000 0001 ih........I.....
0x0040: 0000 0000 0000 0000 0000 0000 0000 0000 ................
15:02:25.233658 IP (tos 0x0, ttl 63, id 41131, offset 0, flags [DF], proto TCP (6), length 80)
10.0.0.252.2049 > 10.0.4.242.49762: Flags [P.], cksum 0x526b (correct), seq 1:29, ack 45, win 252, options [nop,nop,TS val 1826134338 ecr 1768478987], length 28: NFS reply xid 3534571951 reply ok 24 null
0x0000: 4500 0050 a0ab 4000 3f06 810f 0a00 00fc E..P..@.?.......
0x0010: 0a00 04f2 0801 c262 129d 2b1b d8c7 435b .......b..+...C[
0x0020: 8018 00fc 526b 0000 0101 080a 6cd8 9942 ....Rk......l..B
0x0030: 6968 d90b 8000 0018 d2ad 49af 0000 0001 ih........I.....
0x0040: 0000 0000 0000 0000 0000 0000 0000 0000 ................
15:02:25.233662 IP (tos 0x0, ttl 63, id 41131, offset 0, flags [DF], proto TCP (6), length 80)
10.0.0.252.2049 > 10.0.4.242.49762: Flags [P.], cksum 0x526b (correct), seq 1:29, ack 45, win 252, options [nop,nop,TS val 1826134338 ecr 1768478987], length 28: NFS reply xid 3534571951 reply ok 24 null
0x0000: 4500 0050 a0ab 4000 3f06 810f 0a00 00fc E..P..@.?.......
0x0010: 0a00 04f2 0801 c262 129d 2b1b d8c7 435b .......b..+...C[
0x0020: 8018 00fc 526b 0000 0101 080a 6cd8 9942 ....Rk......l..B
0x0030: 6968 d90b 8000 0018 d2ad 49af 0000 0001 ih........I.....
0x0040: 0000 0000 0000 0000 0000 0000 0000 0000 ................
15:02:25.233760 IP (tos 0x0, ttl 64, id 36898, offset 0, flags [DF], proto TCP (6), length 52)
10.0.4.242.49762 > 10.0.0.252.2049: Flags [.], cksum 0x1a14 (incorrect -> 0xeee7), seq 45, ack 29, win 254, options [nop,nop,TS val 1768478987 ecr 1826134338], length 0
0x0000: 4500 0034 9022 4000 4006 90b4 0a00 04f2 E..4."@.@.......
0x0010: 0a00 00fc c262 0801 d8c7 435b 129d 2b37 .....b....C[..+7
0x0020: 8010 00fe 1a14 0000 0101 080a 6968 d90b ............ih..
0x0030: 6cd8 9942 l..B
15:02:25.233910 IP (tos 0x0, ttl 64, id 36899, offset 0, flags [DF], proto TCP (6), length 52)
10.0.4.242.49762 > 10.0.0.252.2049: Flags [F.], cksum 0x1a14 (incorrect -> 0xeee6), seq 45, ack 29, win 254, options [nop,nop,TS val 1768478987 ecr 1826134338], length 0
0x0000: 4500 0034 9023 4000 4006 90b3 0a00 04f2 E..4.#@.@.......
0x0010: 0a00 00fc c262 0801 d8c7 435b 129d 2b37 .....b....C[..+7
0x0020: 8011 00fe 1a14 0000 0101 080a 6968 d90b ............ih..
0x0030: 6cd8 9942 l..B
15:02:25.234214 IP (tos 0x0, ttl 64, id 41132, offset 0, flags [DF], proto TCP (6), length 52)
10.0.0.252.2049 > 10.0.4.242.49762: Flags [F.], cksum 0xeee6 (correct), seq 29, ack 46, win 252, options [nop,nop,TS val 1826134339 ecr 1768478987], length 0
0x0000: 4500 0034 a0ac 4000 4006 802a 0a00 00fc E..4..@.@..*....
0x0010: 0a00 04f2 0801 c262 129d 2b37 d8c7 435c .......b..+7..C\
0x0020: 8011 00fc eee6 0000 0101 080a 6cd8 9943 ............l..C
0x0030: 6968 d90b ih..
15:02:25.234261 IP (tos 0x0, ttl 63, id 41132, offset 0, flags [DF], proto TCP (6), length 52)
10.0.0.252.2049 > 10.0.4.242.49762: Flags [F.], cksum 0xeee6 (correct), seq 29, ack 46, win 252, options [nop,nop,TS val 1826134339 ecr 1768478987], length 0
0x0000: 4500 0034 a0ac 4000 3f06 812a 0a00 00fc E..4..@.?..*....
0x0010: 0a00 04f2 0801 c262 129d 2b37 d8c7 435c .......b..+7..C\
0x0020: 8011 00fc eee6 0000 0101 080a 6cd8 9943 ............l..C
0x0030: 6968 d90b ih..
15:02:25.234262 IP (tos 0x0, ttl 63, id 41132, offset 0, flags [DF], proto TCP (6), length 52)
10.0.0.252.2049 > 10.0.4.242.49762: Flags [F.], cksum 0xeee6 (correct), seq 29, ack 46, win 252, options [nop,nop,TS val 1826134339 ecr 1768478987], length 0
0x0000: 4500 0034 a0ac 4000 3f06 812a 0a00 00fc E..4..@.?..*....
0x0010: 0a00 04f2 0801 c262 129d 2b37 d8c7 435c .......b..+7..C\
0x0020: 8011 00fc eee6 0000 0101 080a 6cd8 9943 ............l..C
0x0030: 6968 d90b ih..
15:02:25.234357 IP (tos 0x0, ttl 64, id 36900, offset 0, flags [DF], proto TCP (6), length 52)
10.0.4.242.49762 > 10.0.0.252.2049: Flags [.], cksum 0x1a14 (incorrect -> 0xeee3), seq 46, ack 30, win 254, options [nop,nop,TS val 1768478988 ecr 1826134339], length 0
0x0000: 4500 0034 9024 4000 4006 90b2 0a00 04f2 E..4.$@.@.......
0x0010: 0a00 00fc c262 0801 d8c7 435c 129d 2b38 .....b....C\..+8
0x0020: 8010 00fe 1a14 0000 0101 080a 6968 d90c ............ih..
0x0030: 6cd8 9943 l..C
15:02:25.234533 IP (tos 0x0, ttl 64, id 28657, offset 0, flags [DF], proto UDP (17), length 68)
10.0.4.242.1004 > 10.0.0.252.20048: [bad udp cksum 0x1a2f -> 0x23cf!] UDP, length 40
0x0000: 4500 0044 6ff1 4000 4011 b0ca 0a00 04f2 E..Do.@.@.......
0x0010: 0a00 00fc 03ec 4e50 0030 1a2f 9ab6 4e33 ......NP.0./..N3
0x0020: 0000 0000 0000 0002 0001 86a5 0000 0003 ................
0x0030: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0040: 0000 0000 ....
15:02:25.234978 IP (tos 0x0, ttl 64, id 8996, offset 0, flags [DF], proto UDP (17), length 52)
10.0.0.252.20048 > 10.0.4.242.1004: [udp sum ok] UDP, length 24
0x0000: 4500 0034 2324 4000 4011 fda7 0a00 00fc E..4#$@.@.......
0x0010: 0a00 04f2 4e50 03ec 0020 aa99 9ab6 4e33 ....NP........N3
0x0020: 0000 0001 0000 0000 0000 0000 0000 0000 ................
0x0030: 0000 0000 ....
15:02:25.235033 IP (tos 0x0, ttl 63, id 8996, offset 0, flags [DF], proto UDP (17), length 52)
10.0.0.252.20048 > 10.0.4.242.1004: [udp sum ok] UDP, length 24
0x0000: 4500 0034 2324 4000 3f11 fea7 0a00 00fc E..4#$@.?.......
0x0010: 0a00 04f2 4e50 03ec 0020 aa99 9ab6 4e33 ....NP........N3
0x0020: 0000 0001 0000 0000 0000 0000 0000 0000 ................
0x0030: 0000 0000 ....
15:02:25.235036 IP (tos 0x0, ttl 63, id 8996, offset 0, flags [DF], proto UDP (17), length 52)
10.0.0.252.20048 > 10.0.4.242.1004: [udp sum ok] UDP, length 24
0x0000: 4500 0034 2324 4000 3f11 fea7 0a00 00fc E..4#$@.?.......
0x0010: 0a00 04f2 4e50 03ec 0020 aa99 9ab6 4e33 ....NP........N3
0x0020: 0000 0001 0000 0000 0000 0000 0000 0000 ................
0x0030: 0000 0000 ....
15:02:25.235253 IP (tos 0x0, ttl 64, id 28658, offset 0, flags [DF], proto UDP (17), length 192)
10.0.4.242.1004 > 10.0.0.252.20048: [bad udp cksum 0x1aab -> 0x17b9!] UDP, length 164
0x0000: 4500 00c0 6ff2 4000 4011 b04d 0a00 04f2 E...o.@[email protected]....
0x0010: 0a00 00fc 03ec 4e50 00ac 1aab 9bb6 4e33 ......NP......N3
0x0020: 0000 0000 0000 0002 0001 86a5 0000 0003 ................
0x0030: 0000 0001 0000 0001 0000 0048 0000 0000 ...........H....
0x0040: 0000 002d 766d 392d 6e6f 6465 322e 3233 ...-vm9-node2.23
0x0050: 2d31 3036 2d36 312d 3139 332e 6c6f 6e2d -106-61-193.lon-
0x0060: 3031 2e75 6b2e 6170 7073 6f6c 6f2e 636f 01.uk.appsolo.co
0x0070: 6d00 0000 0000 0000 0000 0000 0000 0001 m...............
0x0080: 0000 0000 0000 0000 0000 0000 0000 0030 ...............0
0x0090: 2f65 7870 6f72 742f 7076 632d 6161 6430 /export/pvc-aad0
0x00a0: 6131 3964 2d38 6366 612d 3433 3434 2d62 a19d-8cfa-4344-b
0x00b0: 3538 632d 3539 6639 3561 6366 6363 6637 58c-59f95acfccf7
15:02:25.235818 IP (tos 0x0, ttl 64, id 8997, offset 0, flags [DF], proto UDP (17), length 96)
10.0.0.252.20048 > 10.0.4.242.1004: [udp sum ok] UDP, length 68
0x0000: 4500 0060 2325 4000 4011 fd7a 0a00 00fc E..`#%@[email protected]....
0x0010: 0a00 04f2 4e50 03ec 004c 759b 9bb6 4e33 ....NP...Lu...N3
0x0020: 0000 0001 0000 0000 0000 0000 0000 0000 ................
0x0030: 0000 0000 0000 0000 0000 001c 4300 0001 ............C...
0x0040: 1645 9300 0000 e803 0000 8187 0000 0000 .E..............
0x0050: 0000 00c4 2cf1 b000 0000 0001 0000 0001 ....,...........
15:02:25.235864 IP (tos 0x0, ttl 63, id 8997, offset 0, flags [DF], proto UDP (17), length 96)
10.0.0.252.20048 > 10.0.4.242.1004: [udp sum ok] UDP, length 68
0x0000: 4500 0060 2325 4000 3f11 fe7a 0a00 00fc E..`#%@.?..z....
0x0010: 0a00 04f2 4e50 03ec 004c 759b 9bb6 4e33 ....NP...Lu...N3
0x0020: 0000 0001 0000 0000 0000 0000 0000 0000 ................
0x0030: 0000 0000 0000 0000 0000 001c 4300 0001 ............C...
0x0040: 1645 9300 0000 e803 0000 8187 0000 0000 .E..............
0x0050: 0000 00c4 2cf1 b000 0000 0001 0000 0001 ....,...........
15:02:25.235865 IP (tos 0x0, ttl 63, id 8997, offset 0, flags [DF], proto UDP (17), length 96)
10.0.0.252.20048 > 10.0.4.242.1004: [udp sum ok] UDP, length 68
0x0000: 4500 0060 2325 4000 3f11 fe7a 0a00 00fc E..`#%@.?..z....
0x0010: 0a00 04f2 4e50 03ec 004c 759b 9bb6 4e33 ....NP...Lu...N3
0x0020: 0000 0001 0000 0000 0000 0000 0000 0000 ................
0x0030: 0000 0000 0000 0000 0000 001c 4300 0001 ............C...
0x0040: 1645 9300 0000 e803 0000 8187 0000 0000 .E..............
0x0050: 0000 00c4 2cf1 b000 0000 0001 0000 0001 ....,...........
I was testing it with cilium in kube-proxy free mode, it was working fine in my cluster. Do you use kube-proxy?
@kvaps I'm not using kube-proxy, however I've found if I force delete the pods using the pvs and force unmount the NFS mounts, it works.
Were you deleting the pods consuming the pvs? I was trying to avoid this.
No it was working to me without any additional removings
@kvaps Could you share your mount options and any other server options? It looks like for some reason the mounts are not timing out, they're just stuck.
Unfortunately I have no this setup anymore. The behavior reminds me of my old problem, but you don't use kube-proxy and IPVS though