medperf icon indicating copy to clipboard operation
medperf copied to clipboard

[FEATURE] Make --model and --evaluation mlcubes really optional during `medperf test run`

Open VukW opened this issue 1 year ago • 0 comments

If you are a benchmark owner, you have to implement three cubes: data_prep, model and evaluation. The issue is that you cannot run (or implement) them one-by-one: if the benchmark is not submitted still, you have to pass all three mlcubes before able to run any of them.

I propose to make --model and --evaluation cubes totally optional during medperf test run so Benchmark Owner can start implementing just from data-prep cube, not worrying about other parts, thus develop cubes one by one.

VukW avatar Feb 13 '24 15:02 VukW