onnxruntime
onnxruntime copied to clipboard
microsoft
[oneDNN EP] Improves threading performance and adds support ORT and Eigen threadpools
-
Added oneDNN support to use an unified threadpool with ORT, users can select this with the --dnnl_thread ort build flag.
-
Fixed oversubscription happening with ORT and OpenMP oneDNN threads.
-
Added OrtDnnlProviderOptions struct to support provider creation with new features and customization.
-
Added option for users to customize the numbers of OpenMP and Eigen threads used by oneDNN.
-
perf_test and MNIST training now supports OpenMP, Eigen and ORT threadpools.
-
Java and Python support for new threadpool options.
Description: Previously OneDNN ep suffered from thread oversubscription due to ORT creating it's own thread pool and OneDNN creating another one, this update aims to fix this issue by using one unified threadpool for both the default provider and oneDNN ep by default. Also enables support for Eigen threadpool and prevents the oversubscription issue when using OpenMP. The OrtDnnlProviderOptions struct is added, this allows for deeper customization of the oneDNN ep with features that will be coming in the next few moths.
Motivation and Context
-
Why is this change required? What problem does it solve? Performance related fix, prevents thread oversubscription, improves performance on CPU and allows the user to customize thread distribution according to their workload.
-
If it fixes an open issue, please link to the issue here.
@jywu-msft maintainer please run /azp run Linux DNNL CI Pipeline since this is the primary pipeline for testing the OneDNN execution provider.
@jywu-msft could you please run the CI /azp run Linux DNNL CI Pipline for this PR?
![azure-pipelines[bot] avatar](https://avatars.githubusercontent.com/in/9426?v=4 "Published by a pub.dev verified publisher"){kind=small}
@jywu-msft could you try running /azp run Linux DNNL CI Pipeline again please?
Commenter does not have sufficient privileges for PR 12172 in repo microsoft/onnxruntime
@jywu-msft would you mind running the /azp run Linux DNNL CI Pipeline again?
No pipelines are associated with this pull request.
@jywu-msft Do you know why we got this message?
No pipelines are associated with this pull request.
@jywu-msft Do you know why we got this message?
sorry, i did a copy and paste and it seems the source had a typo.
@jywu-msft could you try running /azp run Linux DNNL CI Pipeline again please?