datashim icon indicating copy to clipboard operation
datashim copied to clipboard
verified publisher icon datashim-io

Download speed from internet into s3 PVC on EKS is slow

Open mahi402 opened this issue 2 years ago • 9 comments

Please find logs, bucket is same region

Defaulted container "driver-registrar" out of: driver-registrar, csi-s3 I0330 18:04:33.986814 1 main.go:164] Version: v2.3.0 I0330 18:04:33.986839 1 main.go:165] Running node-driver-registrar in mode=registration I0330 18:04:33.987521 1 main.go:189] Attempting to open a gRPC connection with: "/csi/csi.sock" I0330 18:04:33.987541 1 connection.go:154] Connecting to unix:///csi/csi.sock I0330 18:04:35.979600 1 main.go:196] Calling CSI driver to discover driver name I0330 18:04:35.979628 1 connection.go:183] GRPC call: /csi.v1.Identity/GetPluginInfo I0330 18:04:35.979632 1 connection.go:184] GRPC request: {} I0330 18:04:35.984351 1 connection.go:186] GRPC response: {"name":"ch.ctrox.csi.s3-driver","vendor_version":"v1.1.1"} I0330 18:04:35.984410 1 connection.go:187] GRPC error: I0330 18:04:35.984416 1 main.go:206] CSI driver name: "ch.ctrox.csi.s3-driver" I0330 18:04:35.984434 1 node_register.go:52] Starting Registration Server at: /registration/ch.ctrox.csi.s3-driver-reg.sock I0330 18:04:35.984594 1 node_register.go:61] Registration Server started at: /registration/ch.ctrox.csi.s3-driver-reg.sock I0330 18:04:35.984645 1 node_register.go:91] Skipping healthz server because HTTP endpoint is set to: "" I0330 18:04:36.393034 1 main.go:100] Received GetInfo call: &InfoRequest{} I0330 18:04:36.393226 1 main.go:107] "Kubelet registration probe created" path="/var/lib/kubelet/plugins/csi-s3/registration" I0330 18:04:36.427337 1 main.go:118] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}

mahi402 avatar Apr 03 '23 16:04 mahi402

@mahi402 thanks but could you paste the logs of csi-s3 container instead

srikumar003 avatar Apr 04 '23 12:04 srikumar003

logscontainer2.txt logscontainer.txt

Please find logs of 2 containers running under deamon set

mahi402 avatar Apr 04 '23 16:04 mahi402

could you please let me know if you are able to go through above logs?

mahi402 avatar Apr 06 '23 20:04 mahi402

if i upload any file from s3 pvc to s3, its very fast, but downloading from internet to s3 pvc slow

mahi402 avatar Apr 07 '23 18:04 mahi402

can you please let me know what type of driver you are using?..is it Rclone, we want to enable caching mechanism for faster speed

mahi402 avatar Apr 14 '23 19:04 mahi402

@mahi402 I looked through your logfiles and it seems one of your csi-s3 daemons (in logscontainer2.txt) is not able to connect to S3 endpoint while the other (ip-10-0-1-25.us-west-2.compute.internal). You could try restarting the csi-s3 daemon pod on the malfunctioning node.

We use goofys as the mounter by default but there is a backend implementation for rclone that is not tested yet.

srikumar003 avatar Apr 15 '23 10:04 srikumar003

thank you for reply, in your opinion if we use rclone, will the performance faster??, and can you please let me know how can we convert this dlf.yaml, to use rclone?

mahi402 avatar Apr 15 '23 13:04 mahi402

HI, also we are getting high data tansfer costs in AWS, if we use csi s3 dataset as pvc to the EKS. , can you please let me know if any alternative implementation?

mahi402 avatar Apr 15 '23 20:04 mahi402

@mahi402 csi-s3 should not make any decisions that impact transfer costs. Can you please try accessing the bucket in the cluster with aws s3 command with the same endpoints as you have configured for the dataset, and inform if the transfer costs are different in this case ? Thanks

srikumar003 avatar Apr 18 '23 23:04 srikumar003