onnx
Clarify specification of mixed precision Pow op.
At some point, the Pow op was updated to allow the base and exponent to be of different types. But the result is of the same type as the base.
It is unclear what is meant by Pow(int(2), float(0.5)).
At least two interpretations are possible when the base is an int, and the exponent is a float:
(a) Cast the exponent to int, before performing the exponentiation, or (b) Cast the base to float, perform a float exponentiation, and cast the result back to int.
A third possibility, which involves a change in the op's type-spec is: (c) Change the return type to be float, and do the obvious computation.
Arguably, usage-wise, (c) is better than (b) is better than (a). But some implementations do assume (a) as the spec.
@copilot please update the Pow spec to specify (b).
https://github.com/onnx/onnx/issues/5852#issuecomment-2922537112 has an example of this capability being used in the wild and a note on the ONNX Runtime behavior, which aligns with (b).
It looks like this is a bit more complicated than I thought. I thought it was only a case of (int, float), in which case we cast both to float, do the pow, and cast to output type. But it looks like we can have all kinds of types, including, for example, (int64 and float16) or float32 and float16. So, we need a more clear formulation.
Always cast to type of base or exponent won't work. We want pow(int, float) as well as pow(float, int) to use float.
We should say cast both to higher precision ... but what about (bfloat16, float16)? A few cases (which won't arise in practice!) are a bit tricky.
We should say cast both to higher precision ... but what about (bfloat16, float16)? A few cases (which won't arise in practice!) are a bit tricky.
PyTorch produces float32 output when the base or exponent are a combination of these two (in either order). This makes sense as float32 combines the precision and dynamic range of both types.
I note that PyTorch does the same for other operations like Add/Sub etc. whereas ONNX does not allow implicit casting for those ops. It was possibly an unintended mistake to allow this inconsistency for Pow?