bee
bee copied to clipboard
refactor: sharky cleanup and simplification of code
Checklist
- [x] I have read the coding guide.
- [ ] My change requires a documentation update, and I have done it.
- [x] I have added tests to cover my changes.
- [ ] I have filled out the description and linked the related issues.
Description
Write operations now follow a very basic round robin strategy with circular distribution of requests.
Open API Spec Version Changes (if applicable)
Motivation and Context (Optional)
Related Issue (Optional)
Screenshots (if appropriate):
This sharky implementation does not have guaranties which shard and location is returned next. As an outcome this could lead to 1) probably more cache invalidation (on hdd/ssd level); and 2) fragmented data. Would it make sense to have sharky to always return first available location in consistent order? This way disk IO will probably hit same sector(s) on disk (probably having cache hit).
This sharky implementation does not have guaranties which shard and location is returned next. As an outcome this could lead to 1) probably more cache invalidation (on hdd/ssd level); and 2) fragmented data. Would it make sense to have sharky to always return first available location in consistent order? This way disk IO will probably hit same sector(s) on disk (probably having cache hit).
Order of writes does not guarantee order of reads however. We cannot tell which chunk will be retrieved in the future, so it does not matter which shard we place them. That being said, there is a some benefit in having shards store only a certain PO. For example, when the node is calculating the reserve sample and is going through the entire reserve, reading the shards in order could be a great optimization.
Consider this again after the major localstore releases.