pranadb icon indicating copy to clipboard operation
pranadb copied to clipboard

Infinite storage

Open purplefox opened this issue 2 years ago • 0 comments

We should allow for older data (cold-set) to be offloaded to S3 buckets (when running on AWS), this would allow Prana dataset to scale to virtually unlimited sizes. Blocks of data would be read and asynchronously pushed while keeping a hot-set local. This would also allow for smaller local storage to be used, reducing costs. Both Kafka (Confluent) and Red Panda do something similar.

purplefox avatar Sep 02 '22 15:09 purplefox