python_mozetl icon indicating copy to clipboard operation
python_mozetl copied to clipboard

Write taar outputs to GCS

Open sunahsuh opened this issue 4 years ago • 0 comments

The current way of passing creds in airflow for taar jobs is not ideal – the cleanest way to do this right now is probably to write to a GCS bucket for all the outputs, and sync the bucket to AWS using the GoogleCloudStorageToS3Operator

Most of the changes are isolated to the s3-interfacing methods here: https://github.com/mozilla/python_mozetl/blob/master/mozetl/taar/taar_utils.py

sunahsuh avatar Oct 30 '19 22:10 sunahsuh