embeddings-benchmark
Allow Numpy >=2.0
Checklist
- [x] Run tests locally to make sure nothing is broken using
make test.
$ make test
...
=================================================================================== 944 passed, 236 skipped, 287 warnings in 146.92s (0:02:26) ====================================================================================
- [x] Run the formatter to format the code using
make lint.
~/dev/mteb> make lint
--- 🧹 Running linters ---
ruff format . # running ruff formatting
695 files left unchanged
ruff check . --fix # running ruff linting
All checks passed!
Fixes #1263
I'm currently running the tests in a docker container python:3.10-bullseye to see what is going wrong with the AUC test
I don't have an x86 machine handy right now, and running with mac ARM emulation is very slow (running for > 30 minutes) I can run later, otherwise somebody else could run this on an x86 machine
The issue is that np.argsort' is stable by default in numpy 2.0 I switched to stable=True` so that results match between numpy 1.0 and 2.0, and between different platforms
Hmm looks like for python 3.9 and 3.10 numpy 2.0.0+ was installed in the tests. For Python 3.8, numpy 1.24 was installed, and "stable" doesn't seem to be a valid keyword for argsort (test).
Looks like numpy 2.0 dropped support for Python 3.8 I can add a wrapper that uses one or the other conditionally
Looks like numpy 2.0 dropped support for Python 3.8 I can add a wrapper that uses one or the other conditionally
I think I am more for dropping support for 3.8 it is getting close to end-of-life anyway (end of this month).
(@Muennighoff looping you in here as well)
Ok, this isn't urgent so I'll probably just wait until Python 3.8 is dropped, then try to merge this as-is without a workaround
@mgraczyk updating from main should resolve outstanding issues and we can merge this in
When I'm trying to run mteb with numpy<2.0, then I got this error TypeError: argsort() got an unexpected keyword argument 'stable'. Maybe bump version in pyproject? @KennethEnevoldsen
Yeah you may need to keep support for <2.0 in some way, because many other libraries that could be used with MTEB do not yet support 2.0 (unstructured, for example)