bigflow
bigflow copied to clipboard
Safe DAGs generation and deployment
There are 2 problems when generating and deploying dags:
- Some workflows can be skipped when generating DAGs, due to errors on module import
- When we are deploying DAGs, we are not checking if there is an image with the proper version. There are 2 common reasons for DAGs and image mismatch – broken/forgotten image build and broken versioning.
First, we should throw an error when we can't import some module, and secondly, we should add a check in the DAGs deployment process, so it can throw an exception if there is no expected image. I'm not sure if we should make it as default (due to backward compatibility).