Anton Lokhmotov
Anton Lokhmotov
Using the [other quantized model](https://zenodo.org/record/3252084/) with `--reverse_input_channels` gives even worse accuracy (14.827%): ``` Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.148 Average Precision (AP) @[...
| Reverse input channels? | Quantized model | Accuracy (mAP) | |-|-|-| | yes | asymmetric | 14.827% | | | symmetric | 15.280% | | no | asymmetric |...
Here's the full conversion log: ``` /usr/bin/python3.6 /home/anton/CK_TOOLS/lib-openvino-gcc-8.4.0-2019_R3.1-linux-64/dldt/model-optimizer/mo.py --model_name converted_model --input_model /home/anton/CK_TOOLS/model-tf-mlperf-ssd-mobilenet-q uantized-finetuned-for.openvino/ssd_mobilenet_v1_quant_ft_no_zero_point_frozen_inference_graph.pb --input_shape [1,300,300,3] --tensorflow_object_detection_api_pipeline_config /home/anton/CK_TOOLS/model-t f-mlperf-ssd-mobilenet-quantized-finetuned-for.openvino/pipeline.config --tensorflow_use_custom_operations_config /home/anton/CK_TOOLS/lib-openvino-gcc-8.4.0-2019_R3.1-linux-64/dldt/model-optimizer/extens ions/front/tf/ssd_v2_support.json Model Optimizer arguments: Common parameters: - Path to the...
After my blunder with #29 where I didn't think of running the ImageNet accuracy script with another data type parameter, I decided to check the COCO accuracy script on the...
For the `ICL-I3-1005G1_OpenVINO-Windows` system, I get the accuracy and the number of lines as expected but also 21248 error lines like: `ERROR: loadgen(800) and payload(2583339204608) disagree on image_idx` for the...
No problems with [DellEMC's results](https://github.com/mlperf/inference_results_v0.5/tree/master/closed/DellEMC/results) (including OpenVINO and TensorRT): ``` anton@velociti:/data/anton/inference_results_v0.5/closed/DellEMC/results$ for mlperf_log_accuracy_json in \ ./DELLEMC_R740xd6248_openvino-linux/ssd-small/Offline/accuracy/mlperf_log_accuracy.json \ ./DELLEMC_R740xd6248_openvino-linux/ssd-small/SingleStream/accuracy/mlperf_log_accuracy.json \ ./DELLEMC_R740xd8276_openvino-linux/ssd-small/Offline/accuracy/mlperf_log_accuracy.json \ ./DELLEMC_R740xd8276_openvino-linux/ssd-small/SingleStream/accuracy/mlperf_log_accuracy.json \ ./R740_T4x4_tensorrt/ssd-small/Server/accuracy/mlperf_log_accuracy.json \ ./R740_T4x4_tensorrt/ssd-small/Offline/accuracy/mlperf_log_accuracy.json \ ; do \...
So the resolution seems to have two aspects to it: - To use the [`pre-release` branch](https://github.com/openvinotoolkit/openvino/tree/pre-release) of OpenVINO, last committed to on 7 October 2019 just before the v0.5 deadline;...
The `2019_R3` release is no good either: ``` $ ck install package --tags=lib,openvino,2019_R3 $ export NPROCS=`grep -c processor /proc/cpuinfo` $ ck benchmark program:mlperf-inference-v0.5 --cmd_key=object-detection --repetitions=1 --skip_print_timers \ --env.CK_LOADGEN_MODE=Accuracy --env.CK_LOADGEN_SCENARIO=Offline \...
... while the benchmark code doesn't build with `2020.1`: ``` $ ck install package --tags=lib,openvino,2020.1 $ export NPROCS=`grep -c processor /proc/cpuinfo` $ ck benchmark program:mlperf-inference-v0.5 --cmd_key=object-detection --repetitions=1 --skip_print_timers \ --env.CK_LOADGEN_MODE=Accuracy...
I'm guessing new include paths need to be set for 2020.1: ``` diff --git a/program/mlperf-inference-v0.5/ov_mlperf_cpu/CMakeLists.txt b/program/mlperf-inference-v0.5/ov_mlperf_cpu/CMakeLists.txt index ab50451..005fd67 100644 --- a/program/mlperf-inference-v0.5/ov_mlperf_cpu/CMakeLists.txt +++ b/program/mlperf-inference-v0.5/ov_mlperf_cpu/CMakeLists.txt @@ -42,6 +42,9 @@ set(OPENVINO_LIBRARY "${OPENVINO_LIB_DIR}/libinference_engine.so") set(OPENVINO_CPU_EXTENSION_LIBRARY...