macOS 'Buggy Accelerate Backend when using numpy 1.19'

[neozenith]

Error Message:

Polyfit sanity test emitted a warning, most likely due to using a buggy Accelerate backend. If you compiled yourself, see site.cfg.example for information. Otherwise report this to the vendor that provided NumPy.

As per the linked issue in numpy, I wanted to raise and document the solution here (or link it to a TROUBLESHOOTING.md) for others trying to use the source code for this book.

https://github.com/numpy/numpy/issues/15947

Current Behavior

git clone https://github.com/practical-nlp/practical-nlp
cd practical-nlp
python3 -m venv .venv
. .venv/bin/activate
python3 -m pip install --upgrade pip setuptools wheel

# This step is now problematic on macOS
python3 -m pip install -r requirements.txt

This will spit errors in scikit-image and gensim about buggy Accelerate backend in numpy 1.19.

Possible Solution

There are two proposed solutions:

1. Fallback to numpy==1.18.0
2. Install openblas

Combining both:

brew install openblas
OPENBLAS="$(brew --prefix openblas)" python3 -m pip install numpy==1.18.0

Updating the pinned version in requirements.txt to numpy==1.18.0 also allows this option:

brew install openblas
OPENBLAS="$(brew --prefix openblas)" python3 -m pip install -r requirements.txt

Context

Why is this a problem?

BLAS stands for Basic Linear Algebra Subprograms. So all the core maths operations in numpy which actually make it fast, can lean on the platform specific implementations of BLAS.

Compiled programs supporting every combination of CPU architecture and OS platform and OS version is a disturbingly difficult task.

So numpy has allowed the OS to provide their own backend for BLAS to delegate this responsibility. What makes this harder is the addition of GPU accelerated versions of BLAS blows out these combinations which make it unwieldy for numpy to support on their own.

So on macOS though, the Accelerate Framework is meant to provide native BLAS support.

numpy being good citizens for the scientific community occasionally need to flag certain versions of these frameworks as bad builds as they produce incorrect results.

That is why we need to swap it out for an alternative in this situation. It isn't the end of the world, just a sharp edge to be aware of in this space.

Conclusion

Hopefully future versions of numpy and Accelarate play nice again and this is not an issue. For now this error will show up because during install, a calculation which is known to trigger the fault in these backends, will raise this error.