dataflow-runner icon indicating copy to clipboard operation
dataflow-runner copied to clipboard

Add ability to auto-discover cluster

Open jbeemster opened this issue 6 years ago • 3 comments

This would be based on name and age where rather than passing the jobflow ID you would pass the jobflow name and dataflow-runner would select the newest cluster available that is running.

Thoughts @alexanderdean @BenFradet ?

jbeemster avatar Feb 13 '18 15:02 jbeemster

I really like this:

  1. It could allow us to avoid hardcoding jobflow IDs (which change as we replace clusters) into DAGs
  2. It allows us to do blue-green switchovers

alexanderdean avatar Feb 13 '18 15:02 alexanderdean

Just another possible way, probably easier to implement: https://github.com/snowplow/dataflow-runner/issues/35

chuwy avatar Feb 13 '18 17:02 chuwy

That could work as well - could be an idea that you could also store a jobflow-id as a Consul KV pair to be checked also...

jbeemster avatar Feb 13 '18 18:02 jbeemster