geesefs icon indicating copy to clipboard operation
geesefs copied to clipboard

There were headers present in the request which were not signed in case of load balancing

Open AlexZIX opened this issue 2 years ago • 2 comments

I use geesefs with Minio. When it works directly with Minio it works fine. But for HA and LB reasons I use cluster of 2 WatchGuard XTM 515 UTM devices. This cluster can balance requests between few servers and works fine for my web services. For ex. I distribute files from my minio servers using it. But when I setup geesefs to use cluster instead of standalone minio server, I got following error when just tries to copy file from mounted drive to local:

Dec 27 15:46:54 wp-angstrem /sbin/geesefs[158010]: main.ERROR Error reading 0 +131072 of webplanner/12/02/1202669.b3db: AccessDenied: There were headers present in the request which were not signed#012#011status code: 400, request id: 16C49D6368BE555C, host id: Dec 27 15:46:54 wp-angstrem /sbin/geesefs[158010]: fuse.ERROR *fuseops.VectoredReadOp error: invalid argument

At the same time I can download it via https or using aws cli. This is mounting record in fstab:

wp3d-archive /archive-s3 fuse.geesefs _netdev,allow_other,--file-mode=0660,--dir-mode=0770,--uid=1001,--gid=1001,--shared-config=/etc/passwd-s3fs,--endpoint=https://s3-cold.bazissoft.ru,--region=ru-central 0 0

It's so interesting that when I copy file to mounted drive and then copy back all works fine. Think because of cashing.

geesefs version 0.30.4

AlexZIX avatar Dec 27 '21 13:12 AlexZIX

Today I've tried to create new bucket for testing and mount it to new VM via load balancer. All works fine. Then I filled it with 500 files with 3 GB total size. It still works fine. Than I dismount this bucket and mount a large bucket (total 750K+ files and 9.4 TB size). And it starts show me such errors. Seems geesfs can't wotk stable with the large buckets. May be I should tune it?

AlexZIX avatar Dec 28 '21 08:12 AlexZIX

You should check that your balancer doesn't add additional headers to proxied requests, because S3 signature algorithm includes headers. Maybe the balancer doesn't add headers for all requests and in that case it works. Try to analyze traffic :-)

vitalif avatar Jan 12 '22 16:01 vitalif