`TestDendrogram` tests fail on non-x86 due to overly strict precision tolerances

[risicle]

Hi,

In the Nix package for seaborn, we're noticing that 4 tests are narrowly failing on aarch64 machines: TestDendrogram.test_ndarray_input, TestDendrogram.test_df_input, TestDendrogram.test_axis0_input & TestDendrogram.test_linkage_scipy. The tests pass with a near-identical setup on x86 systems.

Full build log available at https://hydra.nixos.org/log/mxzg3ykkzdz0q55kwgqnyvz5ixv7m4kl-python3.10-seaborn-0.11.2.drv. This is a build log for linux and you should be able to see all the versions of its relevant dependencies in the log.

In cases like this I think it tends to be something to do with the tests being written expecting x87 extended-precision rounding modes.

[mwaskom]

The odd thing about this is that those tests aren't verifying computation through different means, they're just calculating the attribute the same way as should be happening internally to verify the presence of the attribute and its shape. So without looking into it too closely, I don't really understand why you'd be getting different rounding one way vs the other.

[risicle]

https://en.wikipedia.org/wiki/X87

By default, the x87 processors all use 80-bit double-extended precision internally (to allow sustained precision over many calculations, see IEEE 754 design rationale). A given sequence of arithmetic operations may thus behave slightly differently compared to a strict single-precision or double-precision IEEE 754 FPU.

You didn't do anything special, it's just that you likely generated the "expected values" on an x86 machine.

[mwaskom]

No, I understand that — these tests are not comparing against pre-computed expected values, everything is set up on the fly. e.g. for this test

https://github.com/mwaskom/seaborn/blob/14cd3d338e00aafe009e9518bf8f01090b5fc347/seaborn/tests/test_matrix.py#L496-L500

the expected values are here

https://github.com/mwaskom/seaborn/blob/14cd3d338e00aafe009e9518bf8f01090b5fc347/seaborn/tests/test_matrix.py#L739-L740

and it is testing the outputs of this code

https://github.com/mwaskom/seaborn/blob/14cd3d338e00aafe009e9518bf8f01090b5fc347/seaborn/matrix.py#L625-L628

[mwaskom]

It appears that this might be an unusual issue on a very specific platform (I've never heard of it failing anywhere else) and it looks like you have a workaround so I am going to close with no action; it really doesn't seem to me that seaborn should be doing something different here.