bigflow icon indicating copy to clipboard operation
bigflow copied to clipboard

Safe DAGs generation and deployment

Open beczkowb opened this issue 3 years ago • 0 comments

There are 2 problems when generating and deploying dags:

  1. Some workflows can be skipped when generating DAGs, due to errors on module import
  2. When we are deploying DAGs, we are not checking if there is an image with the proper version. There are 2 common reasons for DAGs and image mismatch – broken/forgotten image build and broken versioning.

First, we should throw an error when we can't import some module, and secondly, we should add a check in the DAGs deployment process, so it can throw an exception if there is no expected image. I'm not sure if we should make it as default (due to backward compatibility).

beczkowb avatar Sep 22 '21 18:09 beczkowb