sparse array length is ambiguous error when indexing list of strings

[samseaver]

I put together the test code below based on a few things I read, and it seems like it would work, but I keep getting this error:

Traceback (most recent call last):
  File "/Users/seaver/Seaver_Lab/Projects/Biochemistry_Functions/./test_neofuzz.py", line 36, in <module>
    neofuzz_process.index(options)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/opt/anaconda3/envs/fuzzy-matching/lib/python3.13/site-packages/neofuzz/process.py", line 79, in index
    self.nearest_neighbours.add_item(i_option, vector)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/fuzzy-matching/lib/python3.13/site-packages/scipy/sparse/_base.py", line 449, in __len__
    raise TypeError("sparse array length is ambiguous; use getnnz()"
                    " or shape[0]")

#!/usr/bin/env python
import random
import string

def generate_random_word(length: int) -> str:
    """This bad boy generates random words of given length"""
    letters = string.ascii_lowercase
    return "".join(random.choice(letters) for _ in range(length))

# Let's generate 20 000 unique random words of length 10 to search in
options = list(set(generate_random_word(length=10) for _ in range(20_000)))

from neofuzz import char_ngram_process
neofuzz_process = char_ngram_process(ngram_range=(1,5), metric="angular", tf_idf=True)
neofuzz_process.index(options)

I've been back and forth but I can't figure out what I'm doing wrong here. I'm using version 0.4 as installed via pip

[raresaxpo]

Agree, I have the same issue. It looks like the neofuzz dependencies are not pinned, and so the latest changes in scikit-learn are breaking changes for neofuzz.. Maybe for now downgrade to 0.3.0

[samseaver]

Just to say that downgrading worked for me, but only if I downgraded my python to version 3.12.x

[x-tabdeveloping]

Hey all, sorry for the radio silence, I have been on holidays and a bit of a burnout break recently. The reason this is happening is because I switched backends in the last version update to Annoy, since there were so many issues with PyNNDescent. The problem is, Annoy doesn't support sparse vectors, so you will either have to convert them to dense before passing them to Neofuzz or running some dimensionality reduction method on them to turn them into dense representations.

I will try to do something about this as soon as I get the time, either by switching to yet another backend, or doing the conversion automatically. I'm sorry for your inconvenience. If you do not experience any problems with the downgraded version, I would suggest using it.