[System Test][5.2.1.0] 4k warp workload fails on power architecture because it does not finish closing the connections it opens

[javieramirez1]

Environment info

• NooBaa Version: VERSION
• Platform: Kubernetes 1.14.1 | minikube 1.1.1 | OpenShift 4.1 | other: specify (Wed Jul 17 16:20:15) spectre17-ib:~/javi # rpm -qa | grep noobaa noobaa-core-5.15.4-20240704.el9.ppc64le (Wed Jul 17 16:20:33) spectre17-ib:~/javi # rpm -qa | grep s3 gpfs.mms3-5.2.1-0.240620.111716.el9.ppc64le

Actual behavior

These are some of the errors that warp shows:

warp: <ERROR> stat error: Head "https://9.11.137.111:6443/warp-new-bucket-july17-put136004/jU%29H3p8O/2.6TtngtTopaorhzK2.rnd": net/http: TLS handshake timeout     warp: <ERROR> stat error: Head "https://9.11.137.111:6443/warp-new-bucket-july17-put136004/%29SjZfVy1/1.Yz6ClpKyiHpKv2eh.rnd": net/http: TLS handshake timeout     warp: <ERROR> stat error: Head "https://9.11.137.111:6443/warp-new-bucket-july17-put136004/a5EsTZf%28/1.1ZG1Kl8WC5yErA6o.rnd": net/http: TLS handshake timeout     warp: <ERROR> upload error:Put "https://9.11.137.111:6443/warp-new-bucket-july17-put136004/7lciZVtW/2.gwV%28ORcMBXJfKbNf.rnd": net/http: TLS handshake timeout

warp: <ERROR> stat error: Head "https://9.11.137.111:6443/warp-new-bucket-july17-put136008/v2k2mEVX/14.v9S%28M7O3Kry0LZ63.rnd": dial tcp 9.11.137.111:6443: connect: connection refused

warp: <ERROR> download error:Get "https://9.11.137.111:6443/warp-new-bucket-july17-put136008/z2Nk6mLh/12.RXYxxzhm6og9f%28lN.rnd": dial tcp 9.11.137.111:6443: connect: connection refused

warp: <ERROR> stat error: Head "https://9.11.137.111:6443/warp-new-bucket-july17-put136008/3AvxGJvY/12.D1qnR26PUXka0xLA.rnd": dial tcp 9.11.137.111:6443: connect: connection refused

from part of noobaa this is what is observed:

Jul 17 16:24:59 spectre17-ib [3569206]: [nsfs/3569206] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><Resource>/warp-new-bucket-july17-put136009/3MIokTKq/18.bcpzd%28XmZEc0tLRe.rnd</Resource><RequestId>lyqgzjes-573dao-d68</RequestId></Error> GET /warp-new-bucket-july17-put136009/3MIokTKq/18.bcpzd%28XmZEc0tLRe.rnd {“host”:“9.11.137.111:6443",“user-agent”:“MinIO (linux; amd64) minio-go/v7.0.66 warp/0.7.7",“authorization”:“AWS4-HMAC-SHA256 Credential=UjPPwl2VEszSu9Jh9nlw/20240717/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=7bb5f2749991b05ff3417f489538336f9d41d15b97d2e7262aab1c181e9e9b06",“x-amz-content-sha256”:“e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",“x-amz-date”:“20240717T232457Z”} Error: FileStat: _path=/gpfs/spectrefs_prt/buckets_6009/warp-new-bucket-july17-put136009 cancelled due to ctime change Jul 17 16:25:18 spectre17-ib node[3619437]: [/3619437]  [LOG] CONSOLE:: generate_entropy: error Error: No disk candidates found  at generate_entropy (/usr/local/noobaa-core/src/util/nb_native.js:138:27)  at async init_rand_seed (/usr/local/noobaa-core/src/util/nb_native.js:63:5) (Wed Jul 17 16:28:28) spectre17-ib:/var/log #

Expected behavior

1.wokload completed without problems

Steps to reproduce

1.run 4 instances of warp with 1k connections per instance   warp mixed --host=9.11.137.111:6443 --access-key="$access_key" --secret-key="$secret_key" --obj.size=1k --objects=2000 --duration=10m --disable-multipart --concurrent=1000 --bucket="warp-new-bucket-july17-put13$i" --insecure --tls  &

More information - Screenshots / Logs / Other output

noobaa_events.zip noobaa.zip