Maths tests resurrected (exp and log functions) with bugfixes (take 2)

[LewisGaul]

trafficstars

Reviving https://github.com/ziglang/zig/pull/12355, which revived https://github.com/ziglang/zig/pull/10415 in turn... Hopefully we can get the CI passing this time!

• Add test framework (at lib/std/math/test.zig) for maths functions that take in a float and output a float of the same type, e.g. exp()
• Create tests for all exp/log functions under lib/std/math/test/
  • Sanity tests, taken from libc-test (e.g. exp.h)
  • Special cases (inf, nan, 0, ...)
  • Boundary cases (last value before inf, first subnormal, ...)
• Remove special case tests from function implementation modules to avoid duplication (leave the basic tests intact)
• Bugfixes uncovered by improved testing:
  • exp2_32() flushing to zero too soon
  • expm1_32() giving incorrect results for a certain range of small negative inputs
  • ln64() giving incorrect results for positive subnormal inputs
  • Some functions returning canonical NaN instead of passing through input NaNs unchanged (as per Musl and GCC)
  • log10_64() returning 32-bit inf and NaN, which would be fine as it's cast up, except casting NaN results in failure to pass through the input unchanged

Example output with 'log_level = .info' for debugging (failing test added for illustration):

$./zig test --zig-lib-dir lib lib/std/math/test.zig --test-filter 'exp32() boundary'
Test [1/1] test.exp.test "math.exp32() boundary"... [default] (info):
[default] (info):  IN:  0x42B97217  0x1.72e42ep6      9.27228317e+01
[default] (info): OUT:  0x7F800000  inf               inf
[default] (info): EXP:  0x7F7FFF84  0x1.ffff08p127    3.40279851e+38
FAILURE: expected exp(0x1.72e42ep6)->0x1.ffff08p127, got inf (32-bit)
[default] (info):
[default] (info):  IN:  0x42B17218  0x1.62e43p6       8.87228393e+01
[default] (info): OUT:  0x7F800000  inf               inf
[default] (info):
[default] (info):  IN:  0x7F7FFFFF  0x1.fffffep127    3.40282346e+38
[default] (info): OUT:  0x7F800000  inf               inf
[default] (info):
[default] (info):  IN:  0x00000001  0x0.000002p-126   1.40129846e-45
[default] (info): OUT:  0x3F800000  0x1p0             1.0e+00
[default] (info):
[default] (info):  IN:  0x80000001  -0x0.000002p-126  -1.40129846e-45
[default] (info): OUT:  0x3F800000  0x1p0             1.0e+00
[default] (info):
[default] (info):  IN:  0x00800000  0x1p-126          1.17549435e-38
[default] (info): OUT:  0x3F800000  0x1p0             1.0e+00
[default] (info):
[default] (info):  IN:  0x80800000  -0x1p-126         -1.17549435e-38
[default] (info): OUT:  0x3F800000  0x1p0             1.0e+00
[default] (info):
[default] (info):  IN:  0xC2CFF1B4  -0x1.9fe368p6     -1.03972076e+02
[default] (info): OUT:  0x00000001  0x0.000002p-126   1.40129846e-45
[default] (info):
[default] (info):  IN:  0xC2CFF1B5  -0x1.9fe36ap6     -1.03972084e+02
[default] (info): OUT:  0x00000000  0x0.0p0           0.0e+00
[default] (info):
[default] (info):  IN:  0xC2AEAC4F  -0x1.5d589ep6     -8.73365402e+01
[default] (info): OUT:  0x00800026  0x1.00004cp-126   1.17549967e-38
[default] (info):
[default] (info):  IN:  0xC2AEAC50  -0x1.5d58ap6      -8.73365478e+01
[default] (info): OUT:  0x007FFFE6  0x0.ffffccp-126   1.17549070e-38
[default] (info):
[default] (info): Subtest summary: 10 passed; 1 failed
Test [1/1] test.exp.test "math.exp32() boundary"... FAIL (Failure)
/mnt/c/Users/legaul/repos/zig/lib/std/math/test.zig:96:17: 0x21906a in Testcase(std.math.exp,"exp",f32).run (test)
                return error.TestExpectedEqual;
                ^
/mnt/c/Users/legaul/repos/zig/lib/std/math/test.zig:120:23: 0x217ecd in runTests (test)
    if (failures > 0) return error.Failure;
                      ^
/mnt/c/Users/legaul/repos/zig/lib/std/math/test/exp.zig:78:5: 0x217398 in test.exp.test "math.exp32() boundary" (test)
    try runTests(cases);
    ^
0 passed; 0 skipped; 1 failed.
error: the following test command failed with exit code 1:
lib/std/math/zig-cache/o/7af2224de94b200df723802d7c69b7dc/test /mnt/c/Users/legaul/repos/zig/build/zig

---

Previous suggestions from @matu3ba that I'm considering out of scope of this PR:

• Adding tests for all extreme values (e.g. +/- inf) rather than targeted boundary cases (see https://github.com/ziglang/zig/pull/10415#discussion_r775308449)

• Taking on all Musl tests

2 ideas for discussion

1. port libc-test to zig

• 1.1. Port the test running functions from libc-test, which is the test library of musl.
• 1.2. Use scripts to generate the test code from the musl test macros.
• more code to write, but less dependency on C/clang
• caveat, but actually a feature: Include the comments in the generated test code.

2. vendor libc-test

• 2.1 use zig stubs for the build scripts and test calls
• less code to write, but more dependency on C/clang

[matu3ba]

Can you rebase against master and force push? The CI failures look unrelated to me.

[LewisGaul]

Can you rebase against master and force push? The CI failures look unrelated to me.

Yeah I saw, and will do. However, the windows failures look problematic where the maths function results are off by one bit or the NaN value has a different bit representation (didn't expect to be getting different maths results on different platforms D: ). I'm running on Windows (WSL1 normally), so was going to build natively and try reproducing...

[matu3ba]

@LewisGaul Could you reproduce the issue locally?

windows:

732/1586 test.math.exp32() sanity... FAILURE: expected exp(0x1.288bbcp3)->0x1.4abc8p13, got 0x1.4abc82p13 (32-bit)
FAIL (Failure)
D:\a\1\s\lib\std\math\test.zig:120:23: 0x7ff643e9fc78 in runTests__anon_65151 (test.exe.obj)
    if (failures > 0) return error.Failure;
                      ^
D:\a\1\s\lib\std\math\test\exp.zig:58:5: 0x7ff643ea0057 in test.math.exp32() sanity (test.exe.obj)
    try runTests(cases);
    ^

looks like precision failure.

742/1586 test.math.exp2_64() special... FAILURE: expected exp2(nan)->nan, got nan (64-bit)
FAILURE: expected exp2(-nan)->-nan, got -nan (64-bit)
FAILURE: expected exp2(nan)->nan, got nan (64-bit)
FAILURE: expected exp2(-nan)->-nan, got -nan (64-bit)
FAIL (Failure)
D:\a\1\s\lib\std\math\test.zig:120:23: 0x7ff643ea3279 in runTests__anon_65629 (test.exe.obj)
    if (failures > 0) return error.Failure;
                      ^
D:\a\1\s\lib\std\math\test\exp2.zig:94:5: 0x7ff643ea337c in test.math.exp2_64() special (test.exe.obj)
    try runTests(cases);
    ^

looks like the error messages are messed up or something very weird happened.

750/1586 test.math.ln32() sanity... FAILURE: expected ln(-0x1.0223ap3)->nan, got nan (32-bit)
FAILURE: expected ln(-0x1.0c34b4p3)->nan, got nan (32-bit)
FAILURE: expected ln(-0x1.a206fp2)->nan, got nan (32-bit)
FAILURE: expected ln(-0x1.a05cc8p-2)->nan, got nan (32-bit)
D:\a\1\s\lib\std\math\test.zig:120:23: 0x7ff643ea58d8 in runTests__anon_65811 (test.exe.obj)
    if (failures > 0) return error.Failure;
                      ^
D:\a\1\s\lib\std\math\test\ln.zig:50:5: 0x7ff643ea5b3d in test.math.ln32() sanity (test.exe.obj)
    try runTests(cases);
    ^
751/1586 test.math.ln32() special... FAILURE: expected ln(0x1.5bf0a8p1)->0x1p0, got 0x1.fffffep-1 (32-bit)
FAILURE: expected ln(-0x1p0)->nan, got nan (32-bit)
FAILURE: expected ln(-inf)->nan, got nan (32-bit)
FAIL (Failure)
D:\a\1\s\lib\std\math\test.zig:120:23: 0x7ff643ea5d73 in runTests__anon_65889 (test.exe.obj)
    if (failures > 0) return error.Failure;
                      ^
D:\a\1\s\lib\std\math\test\ln.zig:55:5: 0x7ff643ea5e6c in test.math.ln32() special (test.exe.obj)
    try runTests(cases);
    ^
752/1586 test.math.ln32() boundary... FAILURE: expected ln(-0x0.000002p-126)->nan, got nan (32-bit)
FAILURE: expected ln(-0x1p-126)->nan, got nan (32-bit)
FAIL (Failure)
D:\a\1\s\lib\std\math\test.zig:120:23: 0x7ff643ea6038 in runTests__anon_65896 (test.exe.obj)
    if (failures > 0) return error.Failure;
                      ^
D:\a\1\s\lib\std\math\test\ln.zig:70:5: 0x7ff643ea623b in test.math.ln32() boundary (test.exe.obj)
    try runTests(cases);
    ^
753/1586 test.math.ln64() sanity... FAILURE: expected ln(-0x1.02239f3c6a8f1p3)->nan, got nan (64-bit)
FAILURE: expected ln(-0x1.0c34b3e01e6e7p3)->nan, got nan (64-bit)
FAILURE: expected ln(-0x1.a206f0a19dcc4p2)->nan, got nan (64-bit)
FAILURE: expected ln(-0x1.a05cc754481d1p-2)->nan, got nan (64-bit)
FAILURE: expected ln(-0x1.5b86ea8118a0ep-1)->nan, got nan (64-bit)
FAIL (Failure)
D:\a\1\s\lib\std\math\test.zig:120:23: 0x7ff643ea66f9 in runTests__anon_65910 (test.exe.obj)
    if (failures > 0) return error.Failure;
                      ^

looks like ln, ln32, ln64 are broken.

Same goes for log2, log10.

[LewisGaul]

I have just reproduced this (getting Zig building on Windows was more work than I expected). I'll try to find time to debug... still not sure why it would be any different on different platforms since this is just Zig code...

looks like the error messages are messed up or something very weird happened.

I don't think the error messages are messed up, it's just that expected and actual are compared bitwise, but there are multiple bit representations that are displayed as 'nan'. There are also some non-NaN values that are off in the last bit for some reason.

[andrewrk]

CI checks failing; no update for 30 days.