helm dependency update extremely slow when authenticated with EC2 role

[angeloudy]

I use gitlab-runner to build lots of my helm charts. Lots of the pipeline would run helm dependency update. I recently rebuilt the docker image that does the job. The helm s3 plugin was upgraded to the latest because I didn't specify the version number.

Now the helm dependency update would take 16 minutes to finish, which used to take less than two minutes. After some investigation I found that the slowiness was caused by authenticatint through EC2 role. If I add ~/.aws/credentials to the docker image, helm dependency update finishes instantly.

[ozbillwang]

Something relative with this PR?

https://github.com/hypnoglow/helm-s3/pull/24

aws sdk go latest version is v1.36.25 already:

https://github.com/aws/aws-sdk-go/releases/tag/v1.36.25

ours is

	github.com/aws/aws-sdk-go v1.27.0

https://github.com/hypnoglow/helm-s3/blob/5312896ec0dbfa9dcf6a160e5b65026a24db80f7/go.mod#L11

[ozbillwang]

could you build the plugin by yourself with latest aws sdk go version v1.36.25 and try helm dependency update again on gitlab runner?

[diversario]

I'm not seeing such extreme slowness, but working with S3 repository is noticeably slower - it went from few seconds to ~1 min.

[gokuatkai]

We are noticing an extremely significant slow down on version 0.9.2 and 10.0.0. up to 12times. Our deploy ci-jobs using 0.9.1 and earlier run for about 30s, while 0.9.2+ up to 7min. Job are run on k8s and use the node EC2 role for credentials

could it maybe be due to the new s3 downloader thingy: https://github.com/aws/aws-sdk-go/pull/2823/files ?

[gokuatkai]

I can confirm, that when using the AWS keys of an IAM user directly, the apply is again fast. I guess we need to look closer in this aws sdk lib to find out what is hapening. and I found this: https://github.com/aws/aws-sdk-go/issues/2972

[LuisPiedra]

In our case, a simple helm repo add goes from 400ms in version 0.9.1 to 20s in 0.9.2 and 0.10.0