opencv_zoo
opencv_zoo copied to clipboard
Refactor benchmark configurations and the framework
Benchmark configuration files should be refactored as follows to improve robustness and readability:
Benchmark:
name: str
type: str
data:
path: str # necessary
files: List[str] # necessary and must be array of filenames
sizes: List[Tuple[int, int]] # optional, but must be array of sizes (w, h); Omit to run withuout resizing
metric:
warmup: int
repeat: int
reduction: str # available reduction methods for now: median, gmean
model:
name: str # necessary and must match with the one in wrapper, such as 'YuNet' in face_detection_yunet/yunet.py
backend: int
target: int
other_parameters: ...
Notes:
typeshould be case-insensitive and is used to initialize dataloader and metric. If current dataloaders or metrics do not support the selected type,BaseImageLoaderandBaseMetricwill be initialized.datatypes are based on either_BaseImageLoader, or_BaseVideoLoader.metric:warmupandrepeatdo not work for video-stream input data.modelis moved into the benchmark dictionary.backendandtargetis moved into the model dictionary.
To do:
- Make the initialization of dataloaders and metrics fall back to base dataloader and metric when
typeis not supported. - Set
BaseImageLoaderas base dataloder. - Set
BaseMetricas base metric. - Instantiate
modelinsideBenchmarkclass. - Remove
setBackendandsetTargetfrom wrapper class. - Add parameter
backendandtargetin the constructor of wrapper class. - Add a argument parser to override
Benchmark[model][backend]andBenchmark[model][target]. - Add an example configuration file like the one above to show how to correctly construct a yaml for benchmarking.