Store parquet/CSV dumps on Cloudflare R2

Open ryscheng opened this issue 1 year ago • 6 comments

Feb 22 '24 19:02 ryscheng

CC @ravenac95 apparently Cloudflare R2 has free egress! https://developers.cloudflare.com/r2/pricing/

Feb 22 '24 19:02 ryscheng

related to https://github.com/opensource-observer/oso/issues/919

Feb 22 '24 19:02 ryscheng

Another option if we want IPFS support https://filebase.com/pricing/

Feb 22 '24 19:02 ryscheng

Let's do this monthly for now, we can increase frequency later

Mar 12 '24 18:03 ryscheng

Sharing that the R2 + Filebase combo is really sweet! You can use the same S3 API (probably a Dagster resource) to push data to both. The only requirements for Filebase would be to package all the tables into a car archie though.

Mar 14 '24 21:03 davidgasquez

This might be superceded by a better lakehouse approach https://github.com/opensource-observer/oso/issues/1209

TBD

Aug 21 '24 01:08 ryscheng

Superceded by https://github.com/opensource-observer/oso/issues/4914

Sep 17 '25 03:09 ryscheng