unable to remove invalid block

[daltschu22]

Mountpoint for Amazon S3 version

1.8.0

AWS Region

us-east-1

Describe the running environment

Running on an ec2 with Rocky 8.10

Runs as a systemd service

Mountpoint options

/usr/bin/mount-s3 --read-only --allow-other --file-mode 0555 --dir-mode 0555 --part-size 134217728 --metadata-ttl 300 --cache /opt/mountpoint/cache/<bucket name> --max-cache-size 1024 <bucket name> --prefix .fuse/references_nosymlinks/

What happened?

Mountpoint was running cleanly for months and then hard failed.

Required a fusermount -zu to be able to remount.

Below logs appeared on many different object keys before the crash. All in the same directory in the bucket.

Relevant log output

Sep 29 00:38:10 <host name> mount-s3[15471]: [WARN] mountpoint_s3::prefetch::caching_stream: error reading block from cache cache_key=ObjectId { inner: InnerObjectId { key: "<object key>", etag: ETag("\"51fbfbec40872e0057cd626920cb58e7-24\"") } } block_index=30 range=18874368..85983232 out of 3101804844 error=IoFailure(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" })
Sep 29 00:38:10 <host name> mount-s3[15471]: [WARN] mountpoint_s3::data_cache::disk_data_cache: unable to remove invalid block: Os { code: 2, kind: NotFound, message: "No such file or directory" }
Sep 29 00:38:10 <host name> mount-s3[15471]: [WARN] mountpoint_s3::prefetch::caching_stream: error reading block from cache cache_key=ObjectId { inner: InnerObjectId { key: "<object key>", etag: ETag("\"51fbfbec40872e0057cd626920cb58e7-24\"") } } block_index=30 range=29360128..230686720 out of 3101804844 error=IoFailure(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" })
Sep 29 00:38:27 <host name> mount-s3[15471]: [WARN] mountpoint_s3::data_cache::disk_data_cache: block could not be deserialized: Io(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" })
Sep 29 00:38:27 <host name> mount-s3[15471]: [WARN] mountpoint_s3::prefetch::caching_stream: error reading block from cache cache_key=ObjectId { inner: InnerObjectId { key: "<object key>", etag: ETag("\"51fbfbec40872e0057cd626920cb58e7-24\"") } } block_index=586 range=614465536..2761949184 out of 3101804844 error=InvalidBlockContent
Sep 29 00:38:59 <host name> mount-s3[15471]: [WARN] mountpoint_s3::data_cache::disk_data_cache: unable to remove invalid block: Os { code: 2, kind: NotFound, message: "No such file or directory" }
Sep 29 00:39:15 <host name> systemd[1]: <service name>.service: Main process exited, code=killed, status=6/ABRT
Sep 29 00:39:15 <host name> systemd[1]: <service name>.service: Failed with result 'signal'.