Chronos: chronos benchmark tool

[TheaperDeng]

motivation

This is inspired by https://docs.openvino.ai/latest/openvino_inference_engine_tools_benchmark_tool_README.html.

We need a benchmark tool for developers and users to benchmark the training and inferencing performance on their own platform.

Our users need an easy command to know what their platform can do. Some of our external developers need to user this benchmark tools to get the performance benchmark result. Currently it's only designed for performance rather than accuracy and we can keep iterating on the tool.

usage

benchmark-chronos -model/m <model_name>\ # model name
                    -stage/s <"train"/"latency"/"throuput">\ # train, inference latency, inference throughput
                    -dataset/d <"nyc_taxi"/"electricy">\ # dataset name, currently for built-in dataset
                    -framework/f <"torch"/"tensorflow">\ #framework name(optional), default to "torch" 
                    -cores/c 4\ # core number(optional), default to all physical cores
                    -cpuset/u 0,1,2,3\ # bind to some cpu id(optional), default to no pinning
                    -lookback/l <lookback>\ # lookback(optional), default to dataset suggested lookback
                    -horizon/h <horizon> # horizon(optional), default to dataset suggested horizon

stage 1 Inference

API

benchmark-chronos -model/m <model_name>\ # model name, start from TCN
                    -stage/s <"latency">\ # train, inference latency, inference throughput
                    -dataset/d <"nyc_taxi"/"electricy">\ # dataset name, currently for built-in dataset
                    -framework/f <"torch">\ #framework name(optional), default to "torch" 
                    -cores/c <core num>\ # core number(optional), default to all physical cores
                    -lookback/l <lookback>\ # lookback(optional), default to dataset suggested lookback
                    -horizon/h <horizon> # horizon(optional), default to dataset suggested horizon

An example can be:

benchmark-chronos --model tcn --stages latency --cores 4 --lookback 96 --horizon 720

An example output can be:

>>>>>>>>>>>>> hardware information >>>>>>>>>>>>>
<A lot of hardware information>
<<<<<<<<<<<<< hardware information <<<<<<<<<<<<<

>>>>>>>>>>>>> diagnosis information >>>>>>>>>>>>>
<diagnosis information>
<<<<<<<<<<<<< diagnosis information <<<<<<<<<<<<<

>>>>>>>>>>>>> test-run information >>>>>>>>>>>>>
Model: tcn
Stage: latency
Dataset: nyc_taxi
Cores: 4
lookback: 96
horizon: 720
<<<<<<<<<<<<< diagnosis information <<<<<<<<<<<<<

Running test run...

>>>>>>>>>>>>> test-run result >>>>>>>>>>>>>
avg latency: 5.432ms
90p latency: 5.932ms
95p latency: 6.313ms
99p latency: 6.444ms
<<<<<<<<<<<<< test-run result <<<<<<<<<<<<<