Question about representable values in FP4

[alphaRGB]

Hello, I am curious about FP4 data format. I have seen the binary interchange of FP4 in OCP MXFP4 definition. (OCP Microscaling Formats (MX) Specification Version 1.0)

For FP4 in OCP, only one format E2M1

• Exponent bias: 1
• Infinites: N/A
• NaN: N/A
• Zeros: S 00 0
• Max normal: S 11 1 = 2^2 * 1.5 = 6.0
• Min normal: S 01 0 = 2^0 * 1.0 = 1.0
• Max subnorm: 0.5
• Min subnorm: 0.5

But in your code, the FP4 values = [0, 0.0625, 8.0, 12.0, 4.0, 6.0, 2.0, 3.0, -0, -0.0625, -8.0, -12.0, -4.0, -6.0, -2.0, -3.0]. The max FP4 value in your code is 12.0, it is inconsistent with FP4 in OCP, is there any difference between FP4 encoding/decoding in this project and OCP FP4? https://github.com/TimDettmers/bitsandbytes/blob/e812136c40ef24a02217e6e6c2d03d7c58f0bb03/bitsandbytes/functional.py#L745C5-L745C5

 elif typename == 'fp4':
        # 0b000 = 0
        # 0b001 = 0.0625
        # 0b010 = 8
        # 0b011 = 12
        # 0b100 = 4
        # 0b101 = 6
        # 0b110 = 2
        # 0b111 = 3
        # can also be created with bnb.functional.create_fp8_map(signed=True, exponent_bits=2, precision_bits=1, total_bits=4)
        data = [0, 0.0625, 8.0, 12.0, 4.0, 6.0, 2.0, 3.0, -0, -0.0625, -8.0, -12.0, -4.0, -6.0, -2.0, -3.0]

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[toslunar]

The exponent bias does not matter because the code is normalized by code /= code.max().

However the subnormals are still different (0.0625 is quite small), probably because of a bug in the lines https://github.com/TimDettmers/bitsandbytes/blob/a06a0f6a08cb23754b110359a109e069fa97ce9e/bitsandbytes/functional.py#L279-L284 , where evalue shouldn't be negated.

With

            if evalue == 0:
                # subnormals
                value = value*2**(1-bias)
            else:
                # normals
                value = value*2**(evalue-bias)

(and bias = 2**(exponent_bits-1)-1), the unnormalized data would be OCP FP4

[0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0]

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.