ethereum-etl-airflow icon indicating copy to clipboard operation
ethereum-etl-airflow copied to clipboard

Create CloudStorage classes to abstract file upload/download

Open medvedev1088 opened this issue 5 years ago • 0 comments

Right now we support S3 and GCS as cloud storage providers and file upload/download is done directly in export DAG. Instead 2 classes S3CloudStorage and GCSCloudStorage can be created with common interface to abstract upload/download logic.

As an example:

class S3CloudStorage:
     def download_file(...)
     def upload_file(...)

upload_to_gcs and download_from_gcs can be moved from export DAG to GCSCloudStorage.

medvedev1088 avatar Mar 11 '19 06:03 medvedev1088