rfc: introduce max partition policy to oneDNN Graph API

[ElaineBao]

Description

This RFC is to introduce max partition policy to oneDNN Graph API.

Rendered version: link

[yimeisun123]

1. In the current oneDNN graph implementation, my understanding is that the graph compiler backend already provides additional fusions comparing to DNNL backend. Will specifying "max" policy bring in even more fusions in the graph compiler backend? What happens if we are using graph compiler backend without specifying max policy?
2. It would be good to know how you decide the op list and policy.
3. what would be the case that specifying max policy may have negative impact?

[niuxiaog]

1. In the current oneDNN graph implementation, my understanding is that the graph compiler backend already provides additional fusions comparing to DNNL backend. Will specifying "max" policy bring in even more fusions in the graph compiler backend? What happens if we are using graph compiler backend without specifying max policy?
2. It would be good to know how you decide the op list and policy.
3. what would be the case that specifying max policy may have negative impact?

1. Yes, specifying "max" policy will bring in even more fusions in the graph compiler backend. If we are using graph compiler backend without specifying max policy, the default fusion policy will be used.
2. The op list is roughly the supported ops list of graph compiler.
3. Models in the model zoo will get performance improvement by max policy; For untested models, there may be some regression.

[TaoLv]

@niuxiaog

The op list is roughly the supported ops list of graph compiler.

That's internal. Users don't know which ops are supported by graph compiler from API.

Models in the model zoo will get performance improvement by max policy; For untested models, there may be some regression.

I believe framework integration is for general usage. How can frameworks or framework users know which model is from model zoo and which is not? @chunyuan-w @sanchitintel @yucai-intel @retonym @wenzhe-nrv Please share how you plan to integrate and expose this feature. Thanks.

[sanchitintel]

Hi @ElaineBao, can framework developers tell if custom patterns that don't already exist with fusion policy (but can be added by a backend as predefined patterns) would perform better or worse than max policy?

In other words, would max policy enable better OOB support by not having to request backend teams to add specific pre-defined patterns for new OOB models that would be encountered in the future? Are there any heuristics that can be used to determine if custom pre-defined patterns would perform better or worse than the patterns generated by max policy? Thanks!

[ElaineBao]

Hi, @sanchitintel, I guess you are asking how a FW could determine when to use max and when to use fusion policy.

The general rule is that the default recommended policy is still fusion policy. But for a specific workload that have customed patterns that have not been covered in fusion policy, FW can try to use max policy to see if it achieves better performance.

[TaoLv]

The proposal was reviewed internally and will not be implemented in the library as the use scenario in frameworks is not clear. Thank you all for the discussions and participation.