DirectXTex icon indicating copy to clipboard operation
DirectXTex copied to clipboard

Add FP16 NaN/infinity handling flags

Open elasota opened this issue 6 years ago • 3 comments

Adds 2 flags for more control over special floating point values when converting to FP16. May be useful for applications that depend on NaN/infinity for special handling, like using NaN as a transparency sentinel.

TEX_FILTER_FLOAT_SATURATE_TO_INF saturates out-of-range values to infinity, including existing infinities. TEX_FILTER_FLOAT_KEEP_NANS keeps NaNs instead of forcing them to zero.

Default behavior is convert NaN to zero and saturate to max range.

elasota avatar May 13 '18 18:05 elasota

I'd have implemented these as part of the _ConvertScanline special-case path rather than _LoadScanline/_StoreScanline since it's specific to conversion codepaths. I realize something would need to change in Store with respect to the clamp, but maybe a bool clamp = true parameter would do it.

Since it's specific to FP32 to FP16 conversions, maybe name the flags: TEX_FILTER_FLOAT16_*

walbourn avatar May 14 '18 16:05 walbourn

I totally get what you are going for here, I'm just wondering about some other impacts this might have elsewhere. I'll think about it a bit more...

walbourn avatar May 31 '18 20:05 walbourn

Revisiting this issue part of the problem I think is the functionality offered in CVT16/F16C. These instructions let you control rounding, but don't have any ability to control NANs and/or DENORMS. I of course can implement it in software, but it will slow it down a bit.

As such, I think the default behavior needs to be "use NANs and DENORMs" and provide explicit flags for flushing DENORMs (which would need to be implemented for both F16 and F32 to make sense) and preferring INFs to specials for conversions.

I have some DirectXMath bugs in the handling of NANs. and DENORMS to deal with first, and when I wrap those up I'll revisit this PR.

walbourn avatar Jun 25 '20 21:06 walbourn