dataflow-runner
dataflow-runner copied to clipboard
Add ability to auto-discover cluster
This would be based on name and age where rather than passing the jobflow ID you would pass the jobflow name and dataflow-runner would select the newest cluster available that is running.
Thoughts @alexanderdean @BenFradet ?
I really like this:
- It could allow us to avoid hardcoding jobflow IDs (which change as we replace clusters) into DAGs
- It allows us to do blue-green switchovers
Just another possible way, probably easier to implement: https://github.com/snowplow/dataflow-runner/issues/35
That could work as well - could be an idea that you could also store a jobflow-id as a Consul KV pair to be checked also...