ethereum-etl-airflow
ethereum-etl-airflow copied to clipboard
Create CloudStorage classes to abstract file upload/download
Right now we support S3 and GCS as cloud storage providers and file upload/download is done directly in export DAG. Instead 2 classes S3CloudStorage and GCSCloudStorage can be created with common interface to abstract upload/download logic.
As an example:
class S3CloudStorage:
def download_file(...)
def upload_file(...)
upload_to_gcs
and download_from_gcs
can be moved from export DAG to GCSCloudStorage.