etl
etl copied to clipboard
Add ETL active updates and heartbeats.
Part of m-lab/dev-tracker#501
Use gardener update/ to send per task updates. Use gardener heartbeat/ to send per job heartbeat, once per minute. This allows gardener to detect ETL instance crashes.
Lots of prototyping in the universal-sandbox-gfr branch. Had stuff working with alpine-glibc, but Stephen suggested avoiding that dependency. However, with
FROM ubuntu
RUN apk add --no-cache ca-certificates
I see errors on the etl side like:
active.go:170: Error streaming Get https://storage.googleapis.com/storage/v1/b/archive-mlab-sandbox?alt=json&prettyPrint=false&projection=full: x509: certificate signed by unknown authority
Trying again with:
FROM frolvlad/alpine-glibc
RUN apk update && apk add ca-certificates && rm -rf /var/cache/apk/*