armory
armory copied to clipboard
Partial results and "continue" functionality for interrupted scenarios
Currently, if a scenario is interrupted due to any factor (e.g., OOM or KeyboardInterrupt), no results are written to disk.
This is particularly painful in long-running scenarios that fail several examples into a run, or if a run needs to be stopped for a server reboot.
An alternative would be to stream results to an intermediate file (e.g., write a line before example or batch to file and flush), and then combine the intermediate results to the final results json if the scenario completes successfully.
It would likely require a continue method to do the remaining examples.
MLFlow may handle this for us