seldon-core icon indicating copy to clipboard operation
seldon-core copied to clipboard

fix(Scheduler): No dataflow engines available for terminated pipelines

Open MiguelAAe opened this issue 5 months ago • 1 comments

What this PR does / why we need it: Terminating and terminated pipeline statuses aren't currently saved in the db when a pipeline Scheduler receives an unload request for a given pipeline (its status of deleted true is saved). On a restart the pipeline is restored with status terminate which causes the Scheduler to go through the pipeline state machine of terminate -> terminating -> terminated and since this process is faster than stablishing a connection to Dataflow from the ChainerServer we print no dataflow engines available to handle pipeline.

In short: restarts of the schedulers causes it to print warn logs of no dataflow engines available to handle pipeline of deleted pipelines

Which issue(s) this PR fixes:

Fixes #

  • Suppress warning logs of no dataflow engines available to handle pipelines with status terminated
  • Terminating and Terminated pipeline statuses are now saved in badger db

Tests #

  • creation of pipelines, deletion and restart verified do not produce no dataflow engines available to handle pipeline
  • creation of pipelines, deletion and restart verified do not produce no dataflow engines available to handle pipeline when the scheduler is updated to this fix.

Special notes for your reviewer:

MiguelAAe avatar Jun 11 '25 15:06 MiguelAAe

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Jun 11 '25 15:06 CLAassistant