python_mozetl
python_mozetl copied to clipboard
Write taar outputs to GCS
The current way of passing creds in airflow for taar jobs is not ideal – the cleanest way to do this right now is probably to write to a GCS bucket for all the outputs, and sync the bucket to AWS using the GoogleCloudStorageToS3Operator
Most of the changes are isolated to the s3-interfacing methods here: https://github.com/mozilla/python_mozetl/blob/master/mozetl/taar/taar_utils.py