implement `incbench` command for ease-of-use benchmark

[xin3he]

Type of Change

feature

Description

• [x] implement incbench command as entrypoint for ease-of-use benchmark
• [x] automatically check numa/socket info and dump it with table for ease-of-understand
• [x] supports both Linux and Windows platform
• [x] add benchmark documents
• [x] dump benchmark summary
• [x] add benchmark UTs

General Use Cases

1. incbench main.py: run 1 instance on NUMA:0.
2. incbench --num_i 2 main.py: run 2 instances on NUMA:0.
3. incbench --num_c 2 main.py: run multi-instances with 2 cores per instance on NUMA:0.
4. incbench -C 24-47 main.py: run 1 instance on COREs:24-47.
5. incbench -C 24-47 --num_c 4 main.py: run multi-instances with 4 COREs per instance on COREs:24-47.

Note: > - num_i works the same as num_instances > - num_c works the same as num_cores_per_instance

Expected Behavior & Potential Risk

How has this PR been tested?

tested on Ultra7 1003H (1 socket, 6 P-Cores, 8 E-Cores and 2 SoC E-Cores) and cascade lake(2 numa, 24 cores per numa)

Dependency Change?

N/A

[github-actions[bot]]

⛈️ Required checks status: Has failure 🔴

Warning If you do not have the access to re-run the Probot, please contact XuehaoSun for help. If you push a new commit, all of the workflow will be re-triggered.

Groups summary

🟢 Code Scan Tests workflow

Check ID	Status	Error details
Code-Scan	success		✅
Code-Scan (Bandit Code Scan Bandit)	success		✅
Code-Scan (DocStyle Code Scan DocStyle)	success		✅
Code-Scan (Pylint Code Scan Pylint)	success		✅

These checks are required after the changes to neural_compressor/common/base_config.py, neural_compressor/common/benchmark.py, neural_compressor/common/utils/utility.py, neural_compressor/tensorflow/utils/utility.py, neural_compressor/torch/algorithms/static_quant/utility.py, neural_compressor/torch/utils/utility.py, setup.py.

🟢 Code Scan Neural-Solution Tests workflow

Check ID	Status	Error details
Code-Scan-Neural-Solution	success		✅
Code-Scan-Neural-Solution (Bandit Code Scan Bandit)	success		✅
Code-Scan-Neural-Solution (DocStyle Code Scan DocStyle)	success		✅
Code-Scan-Neural-Solution (Pylint Code Scan Pylint)	success		✅

These checks are required after the changes to setup.py.

🟢 Model Tests workflow

Check ID	Status	Error details
Model-Test	success		✅
Model-Test (Generate Report GenerateReport)	success		✅
Model-Test (Run ONNX Model resnet50-v1-12)	success		✅
Model-Test (Run PyTorch Model resnet18_fx)	success		✅
Model-Test (Run TensorFlow Model resnet50v1.5)	success		✅
Model-Test (Run TensorFlow Model ssd_resnet50_v1)	success		✅

These checks are required after the changes to setup.py.

🟢 Model Tests 3x workflow

Check ID	Status	Error details
Model-Test-3x	success		✅
Model-Test-3x (Generate Report GenerateReport)	success		✅
Model-Test-3x (Run PyTorch Model opt_125m_woq_gptq_int4)	success		✅
Model-Test-3x (Run PyTorch Model opt_125m_woq_gptq_int4_dq_bnb)	success		✅
Model-Test-3x (Run PyTorch Model opt_125m_woq_gptq_int4_dq_ggml)	success		✅

These checks are required after the changes to neural_compressor/common/base_config.py, neural_compressor/common/benchmark.py, neural_compressor/common/utils/utility.py, neural_compressor/torch/algorithms/static_quant/utility.py, neural_compressor/torch/utils/utility.py, setup.py.

🟢 Unit Tests basic workflow

Check ID	Status	Error details
UT-Basic	success		✅
UT-Basic (Coverage Compare CollectDatafiles)	success		✅
UT-Basic (Unit Test FWKs adaptor Test FWKs adaptor)	success		✅
UT-Basic (Unit Test FWKs adaptor baseline Test FWKs adaptor baseline)	success		✅
UT-Basic (Unit Test ITEX Test ITEX)	success		✅
UT-Basic (Unit Test ITEX baseline Test ITEX baseline)	success		✅
UT-Basic (Unit Test Pruning Test PyTorch Pruning)	success		✅
UT-Basic (Unit Test Pruning Test TensorFlow Pruning)	success		✅
UT-Basic (Unit Test Pruning baseline Test PyTorch Pruning baseline)	success		✅
UT-Basic (Unit Test Pruning baseline Test TensorFlow Pruning baseline)	success		✅
UT-Basic (Unit Test TF newAPI Test TF newAPI)	success		✅
UT-Basic (Unit Test TF newAPI baseline Test TF newAPI baseline)	success		✅
UT-Basic (Unit Test User facing API Test User facing API)	success		✅
UT-Basic (Unit Test User facing API baseline Test User facing API baseline)	success		✅
UT-Basic (Unit Test other basic case Test other basic case)	success		✅
UT-Basic (Unit Test other cases baseline Test other cases baseline)	success		✅

UT-Basic coverage report

	Base coverage	PR coverage	Diff
Lines	76.020%	76.096%	0.076%
Branches	65.111%	65.176%	0.065%

These checks are required after the changes to setup.py.

🔴 Unit Tests basic no coverage workflow

Check ID	Status	Error details
UT-Basic-No-Coverage	failure		❌
UT-Basic-No-Coverage (Unit Test FWKs adaptor Test FWKs adaptor)	success		✅
UT-Basic-No-Coverage (Unit Test Pruning Test PyTorch Pruning)	success		✅
UT-Basic-No-Coverage (Unit Test Pruning Test TensorFlow Pruning)	success		✅
UT-Basic-No-Coverage (Unit Test User facing API Test User facing API)	failure	download	❌
UT-Basic-No-Coverage (Unit Test other basic case Test other basic case)	success		✅

These checks are required after the changes to setup.py.

🟢 Unit Tests ITREX workflow

Check ID	Status	Error details
UT-ITREX	success		✅

These checks are required after the changes to setup.py.

🟢 Unit Tests Neural-Solution workflow

Check ID	Status	Error details
UT-Neural-Solution	success		✅

These checks are required after the changes to setup.py.

🟢 Unit Tests Neural-Coder workflow

Check ID	Status	Error details
UT-Coder	success		✅

These checks are required after the changes to setup.py.

🟢 Unit Tests 3x-TensorFlow workflow

Check ID	Status	Error details
UT-3x-TensorFlow	success		✅
UT-3x-TensorFlow (Coverage Compare CollectDatafiles)	success		✅
UT-3x-TensorFlow (Unit Test 3x TensorFlow Unit Test 3x TensorFlow)	success		✅
UT-3x-TensorFlow (Unit Test 3x TensorFlow baseline Unit Test 3x TensorFlow baseline)	success		✅

These checks are required after the changes to neural_compressor/common/base_config.py, neural_compressor/common/benchmark.py, neural_compressor/common/utils/utility.py, neural_compressor/tensorflow/utils/utility.py, setup.py.

🟢 Unit Tests 3x-PyTorch workflow

Check ID	Status	Error details
UT-3x-Torch	success		✅
UT-3x-Torch (Coverage Compare CollectDatafiles)	success		✅
UT-3x-Torch (Unit Test 3x Torch Unit Test 3x Torch)	success		✅
UT-3x-Torch (Unit Test 3x Torch baseline Unit Test 3x Torch baseline)	success		✅

These checks are required after the changes to neural_compressor/common/base_config.py, neural_compressor/common/benchmark.py, neural_compressor/common/utils/utility.py, neural_compressor/torch/algorithms/static_quant/utility.py, neural_compressor/torch/utils/utility.py, test/3x/common/test_benchmark.py, test/3x/common/test_utility.py, setup.py.

🟢 Unit Tests 3x-ONNXRT workflow

Check ID	Status	Error details
UT-3x-ONNXRT	success		✅
UT-3x-ONNXRT (Coverage Compare CollectDatafiles)	success		✅
UT-3x-ONNXRT (Unit Test 3x ONNXRT Unit Test 3x ONNXRT)	success		✅
UT-3x-ONNXRT (Unit Test 3x ONNXRT baseline Unit Test 3x ONNXRT baseline)	success		✅

These checks are required after the changes to neural_compressor/common/base_config.py, neural_compressor/common/benchmark.py, neural_compressor/common/utils/utility.py, setup.py, requirements_ort.txt.

---

Thank you for your contribution! 💜

Note This comment is automatically generated and will be updates every 180 seconds within the next 6 hours. If you have any other questions, contact chensuyue or XuehaoSun for help.

[xin3he]

demo: Multiple Instance Benchmark Summary

[xin3he]

ext test: https://inteltf-jenk.sh.intel.com/job/intel-lpot-validation-top-pr-extension-3x-API/65/

[xin3he]

The final demo:

[xin3he]

ext test: https://inteltf-jenk.sh.intel.com/job/intel-lpot-validation-top-pr-extension-3x-API/66/