armory icon indicating copy to clipboard operation
armory copied to clipboard

Partial results and "continue" functionality for interrupted scenarios

Open davidslater opened this issue 2 years ago • 1 comments

Currently, if a scenario is interrupted due to any factor (e.g., OOM or KeyboardInterrupt), no results are written to disk.

This is particularly painful in long-running scenarios that fail several examples into a run, or if a run needs to be stopped for a server reboot.

An alternative would be to stream results to an intermediate file (e.g., write a line before example or batch to file and flush), and then combine the intermediate results to the final results json if the scenario completes successfully.

It would likely require a continue method to do the remaining examples.

davidslater avatar Mar 23 '22 23:03 davidslater

MLFlow may handle this for us

lcadalzo avatar Dec 05 '22 19:12 lcadalzo