seldon-core
seldon-core copied to clipboard
fix(Scheduler): No dataflow engines available for terminated pipelines
What this PR does / why we need it: Terminating and terminated pipeline statuses aren't currently saved in the db when a pipeline Scheduler receives an unload request for a given pipeline (its status of deleted true is saved). On a restart the pipeline is restored with status terminate which causes the Scheduler to go through the pipeline state machine of terminate -> terminating -> terminated and since this process is faster than stablishing a connection to Dataflow from the ChainerServer we print no dataflow engines available to handle pipeline.
In short: restarts of the schedulers causes it to print warn logs of no dataflow engines available to handle pipeline of deleted pipelines
Which issue(s) this PR fixes:
Fixes #
- Suppress warning logs of no dataflow engines available to handle pipelines with status terminated
- Terminating and Terminated pipeline statuses are now saved in badger db
Tests #
- creation of pipelines, deletion and restart verified do not produce
no dataflow engines available to handle pipeline - creation of pipelines, deletion and restart verified do not produce
no dataflow engines available to handle pipelinewhen the scheduler is updated to this fix.
Special notes for your reviewer: