oso icon indicating copy to clipboard operation
oso copied to clipboard

Published 20 hours ago • verified publisher icon opensource-observer

Import Starboard S3 dumps into BigQuery

Open ryscheng opened this issue 9 months ago • 2 comments

What is it?

We can probably use either the BQ data transfer service or Airbyte. Probably want to do this as a Dagster asset

ryscheng avatar Apr 27 '24 03:04 ryscheng

Even more preferable is a Databricks account like this: https://github.com/davidgasquez/filecoin-data-portal/blob/main/fdp/assets/lily.py https://github.com/davidgasquez/filecoin-data-portal/blob/main/fdp/resources.py

ryscheng avatar Apr 29 '24 23:04 ryscheng

FWIW, it looks like this is built on static data files, https://filecoinpulse.pages.dev/

It'd be cool to see it powered by some live data if we can make this happen

ryscheng avatar Apr 30 '24 00:04 ryscheng

Deprecating in favor of this issue https://github.com/opensource-observer/oso/issues/764

ryscheng avatar Aug 21 '24 01:08 ryscheng