storj
Gateway: add a custom endpoint to retrieve total object count
The endpoint should return values for:
- total object count (includes encrypted + decrypted objects) - this can be done with iteratively calling "libuplink list" with the "encrypted" flag
- encrypted object count
It should also be limited in some way (e.g. if there are 10s of millions of objects). Add a config value.
See "list buckets with attribution" for an example of a custom endpoint on gateway: https://github.com/storj/gateway-mt/blob/267c4fc8f7fdbb1953a8636ea305368a503d5506/pkg/minio/api-response.go#L25
Context for underlying issue: https://github.com/storj/storj/issues/6080
AC:
- new endpoint added to gateway for listing total object counts on the bucket level
- add configuration for restricting this functionality in situations where the object count is very high
![storj-gerrit[bot] avatar](https://avatars.githubusercontent.com/u/6308081?v=4 "Published by a pub.dev verified publisher"){kind=small}
Is there any reason we decided not to add something to the satellite API instead of going through gateway with ListObjects calls? I may be missing some context, but just looking at the gateway change it seems like calling ListObjects might not be an efficient way of doing it.
BTW: it would be helpful if background info is added to the issue and the commit message to aid in reviewing.
@halkyon - The context is in https://github.com/storj/storj/issues/6080 (added to description as well)
@ferristocrat Unfortunately it doesn't have everything, there's at least a couple of Slack threads with more details I had to go hunt down. Part of my problem generally is our reliance on Slack too much (it's horrible as a documentation source, and it's exclusive to open source contributions), but that's for another discussion ;)
Anyway, judging by some discussions there's still some work to figure out what to do on this.
GitHub Thread discussing options: https://storj.slack.com/archives/CAQV0AE2Z/p1698676617000929