gensim

gensim copied to clipboard

Problem description

Run the test for the new version of WEFE

Steps/code/corpus to reproduce

../../../../../../home/raffaele/.virtualenvs/gensim4/lib/python3.8/site-packages/scipy/sparse/sparsetools.py:21
  /home/raffaele/.virtualenvs/gensim4/lib/python3.8/site-packages/scipy/sparse/sparsetools.py:21: DeprecationWarning: `scipy.sparse.sparsetools` is deprecated!
  scipy.sparse.sparsetools is a private module for scipy.sparse, and should not be used.
    _deprecated()

../../../../../../home/raffaele/.virtualenvs/gensim4/lib/python3.8/site-packages/sklearn/linear_model/_least_angle.py:34
  /home/raffaele/.virtualenvs/gensim4/lib/python3.8/site-packages/sklearn/linear_model/_least_angle.py:34: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. Use `float` by itself, which is identical in behavior, to silence this warning. If you specifically wanted the numpy scalar type, use `np.float_` here.
    method='lar', copy_X=True, eps=np.finfo(np.float).eps,

../../../../../../home/raffaele/.virtualenvs/gensim4/lib/python3.8/site-packages/sklearn/linear_model/_least_angle.py:164
  /home/raffaele/.virtualenvs/gensim4/lib/python3.8/site-packages/sklearn/linear_model/_least_angle.py:164: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. Use `float` by itself, which is identical in behavior, to silence this warning. If you specifically wanted the numpy scalar type, use `np.float_` here.
    method='lar', copy_X=True, eps=np.finfo(np.float).eps,

../../../../../../home/raffaele/.virtualenvs/gensim4/lib/python3.8/site-packages/sklearn/linear_model/_least_angle.py:281
  /home/raffaele/.virtualenvs/gensim4/lib/python3.8/site-packages/sklearn/linear_model/_least_angle.py:281: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. Use `float` by itself, which is identical in behavior, to silence this warning. If you specifically wanted the numpy scalar type, use `np.float_` here.
    eps=np.finfo(np.float).eps, copy_Gram=True, verbose=0,

../../../../../../home/raffaele/.virtualenvs/gensim4/lib/python3.8/site-packages/sklearn/linear_model/_least_angle.py:865
  /home/raffaele/.virtualenvs/gensim4/lib/python3.8/site-packages/sklearn/linear_model/_least_angle.py:865: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. Use `float` by itself, which is identical in behavior, to silence this warning. If you specifically wanted the numpy scalar type, use `np.float_` here.
    eps=np.finfo(np.float).eps, copy_X=True, fit_path=True,

../../../../../../home/raffaele/.virtualenvs/gensim4/lib/python3.8/site-packages/sklearn/linear_model/_least_angle.py:1121
  /home/raffaele/.virtualenvs/gensim4/lib/python3.8/site-packages/sklearn/linear_model/_least_angle.py:1121: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. Use `float` by itself, which is identical in behavior, to silence this warning. If you specifically wanted the numpy scalar type, use `np.float_` here.
    eps=np.finfo(np.float).eps, copy_X=True, fit_path=True,

../../../../../../home/raffaele/.virtualenvs/gensim4/lib/python3.8/site-packages/sklearn/linear_model/_least_angle.py:1149
  /home/raffaele/.virtualenvs/gensim4/lib/python3.8/site-packages/sklearn/linear_model/_least_angle.py:1149: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. Use `float` by itself, which is identical in behavior, to silence this warning. If you specifically wanted the numpy scalar type, use `np.float_` here.
    eps=np.finfo(np.float).eps, positive=False):

../../../../../../home/raffaele/.virtualenvs/gensim4/lib/python3.8/site-packages/sklearn/linear_model/_least_angle.py:1379
  /home/raffaele/.virtualenvs/gensim4/lib/python3.8/site-packages/sklearn/linear_model/_least_angle.py:1379: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. Use `float` by itself, which is identical in behavior, to silence this warning. If you specifically wanted the numpy scalar type, use `np.float_` here.
    max_n_alphas=1000, n_jobs=None, eps=np.finfo(np.float).eps,

../../../../../../home/raffaele/.virtualenvs/gensim4/lib/python3.8/site-packages/sklearn/linear_model/_least_angle.py:1621
  /home/raffaele/.virtualenvs/gensim4/lib/python3.8/site-packages/sklearn/linear_model/_least_angle.py:1621: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. Use `float` by itself, which is identical in behavior, to silence this warning. If you specifically wanted the numpy scalar type, use `np.float_` here.
    max_n_alphas=1000, n_jobs=None, eps=np.finfo(np.float).eps,

../../../../../../home/raffaele/.virtualenvs/gensim4/lib/python3.8/site-packages/sklearn/linear_model/_least_angle.py:1755
  /home/raffaele/.virtualenvs/gensim4/lib/python3.8/site-packages/sklearn/linear_model/_least_angle.py:1755: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. Use `float` by itself, which is identical in behavior, to silence this warning. If you specifically wanted the numpy scalar type, use `np.float_` here.
    eps=np.finfo(np.float).eps, copy_X=True, positive=False):

Versions

Please provide the output of:

Linux-5.8.0-33-generic-x86_64-with-glibc2.32
Python 3.8.6 (default, Sep 25 2020, 09:36:53) 
[GCC 10.2.0]
Bits 64
NumPy 1.20.0rc1
SciPy 1.6.0rc1
gensim 4.0.0beta
FAST_VERSION 1

Thanks for trying the beta and reporting!

I don't think we can do much about sparsetools (scipy provides no alternative AFAIK), but we can definitely fix the float.

@raffaem can you run whatever steps you used before, but on the current develop branch of Gensim?

We removed and fixed a bunch of code, so maybe this is not relevant any more. Thanks.

Removing from 4.0.0. Will revisit when @raffaem follows up.

#3066 is related to this

I can confirm that the sparsetools DeprecationWarning still exists in gensim 4.0.1. Would it make sense to just squash this warning by surrounding the import with a try/except? The warning is certainly relevant to gensim maintainers, but for gensim users, it's mainly just an annoyance.

Getting rid of (not showing) the sparsetools warning makes sense. But I don't think try-expect will help here – it's not an exception.

As far as I'm aware scipy doesn't offer any alternative for its deprecated (but still functional) sparsetools module. So "fixing" the warning properly is not possible.

I seem to remember there is a way to achieve this sort of thing

I think using warnings.carch_warnings()

https://stackoverflow.com/questions/9134795/how-to-get-rid-of-specific-warning-messages-in-python-while-keeping-all-other-wa

Update from me: So it turns out that Python's default warning filters will actually filter out DeprecationWarnings, so if you don't for some reason override this, gensim user's won't see these warnings. For some reason though pytest will show these warnings, which is where I was seeing it (see https://stackoverflow.com/a/53218641/896660). I don't really understand why pytest is not using the default warning filters of the python interpreter, but at least there is a way to configure this, so this is the solution in my particular case. I don't think there's anything that needs to be done on gensim's end.

This was previously discussed in #462 (including with scipy upstream devs).

Removing scipy from the dependencies was also discussed in #557.

And in https://github.com/scipy/scipy/issues/5348 by the scipy team.

Scipy (and the whole pydata ecosystem) is much easier to deploy and manage than it was 7 years ago. Plus gensim now compiles its modules too, it's no longer pure python. So many of the assumptions and design edges from the original discussion are no longer relevant.