public-cloud-roadmap icon indicating copy to clipboard operation
public-cloud-roadmap copied to clipboard

Add objects count & bucket volumetry infos in head-bucket response headers

Open lason-ovh opened this issue 10 months ago • 0 comments

OVH current offers for objectstorage, especially the highperf one, are intended to be accessible over S3-compatible API (and they are). Yet aren't these new "S3" offers based, at least for the highperf one, based on openio? So would it be possible to "talk" to those "S3" systems using the Openstack Swift API too? Why am I asking? Openstack Swift has a nice feature : when listing buckets (and I mean getting the list of buckets, not listing the content of a bucket) Swift can easily report a pre-computed, not up-to-date, count of objects and total byte size of the bucket. When an architecture needs to automatically monitor growth, it is much more effective to use Swift API to get those totals, than using S3 API. Because S3 API does offer any provision to read such totals: you have to list objects to count them and add their sizes. With buckets with tens of millions objects, that is completely ridiculous to list objects to count them.

Here is a proposal for a minor alteration to OVH implementation of S3 API, which should not break any S3 compatibility while allow reporting those objectsCount and objectsSize which you (OVH) have access to, as you report them through your own OVH API (which I would like not to depend on for some reason). I would suggest to add two more HTTP headers into the answer to the HEAD / request (see https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html). Typically the answer would have HTTP headers like these:

HTTP/1.1 200
x-amz-bucket-location-type: BucketLocationType
x-amz-bucket-location-name: BucketLocationName
x-amz-bucket-region: BucketRegion
x-amz-access-point-alias: AccessPointAlias
Adding these:
x-ovh-bucket-objects-count: 14285792
x-ovh-bucket-objects-size: 1044701804909

would offer a nice service to specific projects knowing this information is available there, while not impacting S3 compatibility of the API.

lason-ovh avatar Apr 16 '24 12:04 lason-ovh