Jeff Klukas

Seeing this with Beam 2.12 as well.

> I don't know where the requirements file is coming from. It's this line in dataproc_init.sh It looks like it's defined within [telemetry-airflow/dataproc_bootstrap](https://github.com/mozilla/telemetry-airflow/blob/60f7dd1236193da4cfb73ab71210fb129b9638df/dataproc_bootstrap/python-requirements.txt) and the contents of that folder are...

@willkg makes a good point that we could add some additional security around building based on tags via bash logic beyond what's available in CircleCI's config structure directly. I've still...

I do like the idea of having artifacts about what was deployed. Also, the more I think about this the more I dislike the current mismatch between how we deploy...

Related: https://github.com/mozilla/gcp-ingestion/issues/668

Per docs on [PubsubUnboundedSource](https://beam.apache.org/releases/javadoc/2.8.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.html): > Checkpoints are used both to ACK received messages back to Pubsub (so that they may be retired on the Pubsub end), and to NACK already...

There's [a StackOverflow discussion about ack'ing and GroupByKey](https://stackoverflow.com/questions/33362730/shutdown-and-update-job-in-google-dataflow-with-pubsubio-message-guarantees/33375977#33375977) that says something similar. So, it does look like Dataflow in unbounded mode is going to persist messages to intermediate storage (which...

I started a [thread of [email protected]](https://lists.apache.org/thread.html/f2198f28c56c5ef3e7144db11fe80422d2f32fdc417cab1f2668b3bd@%3Cuser.beam.apache.org%3E) about this, with replies from an engineer at Google confirming the problem and indicating that there's no reason Beam couldn't be modified to handle...

There has been no change that I'm aware of in Beam; PubsubIO will still ack messages when they hit a GroupByKey. This is one of a variety of reasons that...

> Anyways, I got here because I was also wondering how can I run my Streaming pipeline without any data loss (in case you want to answer my question) Thanks!...