opencv_zoo icon indicating copy to clipboard operation
opencv_zoo copied to clipboard

Refactor benchmark configurations and the framework

Open fengyuentau opened this issue 4 years ago • 0 comments

Benchmark configuration files should be refactored as follows to improve robustness and readability:

Benchmark:
    name: str
    type: str 
    data:
      path: str # necessary
      files: List[str] # necessary and must be array of filenames
      sizes: List[Tuple[int, int]] # optional, but must be array of sizes (w, h); Omit to run withuout resizing
    metric:
      warmup: int
      repeat: int
      reduction: str # available reduction methods for now: median, gmean
    model:
      name: str # necessary and must match with the one in wrapper, such as 'YuNet' in face_detection_yunet/yunet.py
      backend: int
      target: int
      other_parameters: ...

Notes:

  • type should be case-insensitive and is used to initialize dataloader and metric. If current dataloaders or metrics do not support the selected type, BaseImageLoader and BaseMetric will be initialized.
  • data types are based on either _BaseImageLoader, or _BaseVideoLoader.
  • metric: warmup and repeat do not work for video-stream input data.
  • model is moved into the benchmark dictionary.
    • backend and target is moved into the model dictionary.

To do:

  • Make the initialization of dataloaders and metrics fall back to base dataloader and metric when type is not supported.
  • Set BaseImageLoader as base dataloder.
  • Set BaseMetric as base metric.
  • Instantiate model inside Benchmark class.
  • Remove setBackend and setTarget from wrapper class.
  • Add parameter backend and target in the constructor of wrapper class.
  • Add a argument parser to override Benchmark[model][backend] and Benchmark[model][target].
  • Add an example configuration file like the one above to show how to correctly construct a yaml for benchmarking.

fengyuentau avatar Nov 13 '21 12:11 fengyuentau