💫 Incompatibilities & support on later versions of spaCy

[rmrbytes]

Is there any plan/ date to get neuralcoref to work on the latest version of spaCy (ver 2.1.8)? Thanks

[svlandeg]

Yes, the plan with @honnibal and @ines is to help @thomwolf maintain this awesome spaCy extension and to make sure it can stay in-sync when there are new releases for spaCy. No timeline yet, but stay tuned ;-)

For now, you can work with neuralcoref and spaCy>2.1 if you first install your required version of spaCy, and then build neuralcoref from source. If that doesn't work, you can also try compiling spaCy from source first (in a clean environment), then compile neuralcoref from source.

If you are running into incompatibility issues and prefer not to build from source, the other option is to downgrade spaCy to 2.1.0, which is compatible with neuralcoref 4.0.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[dborncamp]

I believe this is still an issue. Is there another workaround that does not require building from source? Also, I don't want the bot to close this without a fix.

[svlandeg]

No, there's no other workaround for the moment, and yes this issue will be addressed - which is why I removed the wontfix label ;-)

[AlJohri]

For some reason I was having trouble building neuralcoref from source unless I copied the spacy folder into site-packages.

I've attached my reproducible example getting it working with the latest version of spaCy.

Any tips on doing this better / more easily?

all:
	python3 -m venv .venv
	.venv/bin/pip install cython boto3
	git clone https://github.com/explosion/spaCy.git || true
	.venv/bin/pip install -r spaCy/requirements.txt
	.venv/bin/python spaCy/setup.py build_ext --inplace
	cp -R spaCy/spacy .venv/lib/python3.8/site-packages/
	git clone https://github.com/huggingface/neuralcoref.git || true
	.venv/bin/python neuralcoref/setup.py build_ext --inplace
	cp -R neuralcoref/neuralcoref .venv/lib/python3.8/site-packages/
	.venv/bin/python -m spacy download en

test:
	.venv/bin/python -c "import spacy; nlp = spacy.load('en'); import neuralcoref; neuralcoref.add_to_pipe(nlp); doc = nlp(u'My sister has a dog. She loves him.'); print(doc._.coref_clusters)"

clean:
	rm -rf .venv spacy neuralcoref

I really wish I could do something like:

pip install --no-binary --global-option=build_ext --global-option="--inplace" -e spaCy

but that wasn't working for me.

[svlandeg]

The way I do it, after pulling both from github and creating a new venv :

cd spacy
pip install -r requirements.txt
python setup.py build_ext --inplace
pip install -e .

cd ../neuralcoref
python setup.py build_ext --inplace
pip install -e .

[AlJohri]

ahh, I didn't realize you're suppose to pip install after build_ext. thanks!

[svlandeg]

It facilitates things because then spaCy is actually listed in pip list, and you don't have to pip install the requirements.txt for neuralcoref because you should already have everything you need from spaCy. Maybe there's a better way, but this works for me :-)

[AlJohri]

worked for me as well! much nicer than copying things to site-packages. here's the updated version in case anyone needs it:

python3 -m venv .venv
.venv/bin/pip install cython boto3
git clone https://github.com/explosion/spaCy.git || true
.venv/bin/pip install -r spaCy/requirements.txt
.venv/bin/python spaCy/setup.py build_ext --inplace
.venv/bin/pip install -e spaCy
git clone https://github.com/huggingface/neuralcoref.git || true
.venv/bin/python neuralcoref/setup.py build_ext --inplace
.venv/bin/pip install -e neuralcoref
.venv/bin/python -m spacy download en

[arlyon]

In the mean-time, is it perhaps an idea to publish known working combinations of spacy and neuralcoref? The readme currently just states 2.1+.

spacy==4.0.0
neuralcoref==2.1.0

[jonathankingfc]

In the mean-time, is it perhaps an idea to publish known working combinations of spacy and neuralcoref? The readme currently just states 2.1+.

spacy==4.0.0
neuralcoref==2.1.0

I believe this should be the other way around. spaCy latest is 2.2.2 and neuralcoref is 4.0.0. I have just been using

spacy==2.1.3
neuralcoref==4.0.0

I am having issues building from source in my Docker stack. Hopefully this issue can get fixed soon as coreference resolution is something that I am waiting to implement.

[AlJohri]

@svlandeg anything I can do to help here?

[victoriastuart]

Per #222 direct link to comment: https://github.com/huggingface/neuralcoref/issues/222#issuecomment-577952955

BASH:

pip uninstall neuralcoref
git clone https://github.com/huggingface/neuralcoref.git
cd neuralcoref
pip install -r requirements.txt
pip install -e .
pip uninstall spacy
pip install spacy
python -m spacy download en

PYTHON:

import spacy
nlp = spacy.load('en')
import neuralcoref      ## ignore RuntimeWarning(s)
neuralcoref.add_to_pipe(nlp)
doc = nlp(u'My sister has a dog. She loves him.')
doc._.has_coref         ## True
doc._.coref_clusters    ## [My sister: [My sister, She], a dog: [a dog, him]]
doc._.coref_resolved    ## 'My sister has a dog. My sister loves a dog.'

[trias702]

In order to get neuralcoref working with Spacy > 2.2.0, why do you need to build Spacy itself from source? I can understand having to build neuralcoref from source, but why Spacy as well?

[svlandeg]

Because the pip-installed version of spaCy >= 2.2.0 did not distribute the pxd files which neuralcoref builds against. This will be fixed in spaCy in the new versions.

[EDIT: this is indeed fixed in the newer versions of spaCy (e.g. 2.3.2), so building spaCy from source is not required anymore. I updated my original post accordingly to avoid confusion.]

[trias702]

Is it possible to build Spacy > 2.2.0 from source, and then neuralcoref from source, but without using pip install -e?

Ideally, the end goal is to have a wheel distributable neuralcoref which was built against spacy 2.2.3 and which I can deploy to other systems.

I know the following commands work:

cd spacy
pip install -r requirements.txt
python setup.py build_ext --inplace
pip install -e .

cd ../neuralcoref
python setup.py build_ext --inplace
pip install -e .

But how could I correctly accomplish getting a neuralcoref wheel? I'm thinking something like this maybe:

cd spacy
pip install -r requirements.txt
python setup.py build_ext --inplace
python setup.py bdist_wheel
<install the wheel?>

cd ../neuralcoref
python setup.py build_ext --inplace
python setup.py bdist_wheel
<install the wheel?>

Would greatly appreciate any help.

[aced125]

Any update on this?

[svlandeg]

We're currently working hard on getting spaCy v.3 out, which will use the revamped thinc. Once all that is done, the next priority will be to get neuralcoref integrated with the new spaCy.

[aaaailuj]

Is it possible to build Spacy > 2.2.0 from source, and then neuralcoref from source, but without using pip install -e?

Ideally, the end goal is to have a wheel distributable neuralcoref which was built against spacy 2.2.3 and which I can deploy to other systems.

I know the following commands work:

cd spacy
pip install -r requirements.txt
python setup.py build_ext --inplace
pip install -e .

cd ../neuralcoref
python setup.py build_ext --inplace
pip install -e .

But how could I correctly accomplish getting a neuralcoref wheel? I'm thinking something like this maybe:

cd spacy
pip install -r requirements.txt
python setup.py build_ext --inplace
python setup.py bdist_wheel
<install the wheel?>

cd ../neuralcoref
python setup.py build_ext --inplace
python setup.py bdist_wheel
<install the wheel?>

Would greatly appreciate any help.

I did this with

cd ../neuralcoref
python setup.py sdist --formats=gztar

and then

pip install neuralcoref-4.0.tar.gz

[ym-han]

just a quick fyi: I personally still got the <frozen importlib._bootstrap>:219: RuntimeWarning: spacy.morphology.Morphology size changed, may indicate binary incompatibility. Expected 104 from C header, got 112 from PyObject <frozen importlib._bootstrap>:219: RuntimeWarning: spacy.vocab.Vocab size changed, may indicate binary incompatibility. Expected 96 from C header, got 112 from PyObject error, when doing import neuralcoref, even after trying to first install spacy from source and then neuralcoref from source. I'm on a mac, and I think I was on python 3.8.3. The only thing that worked for me was switching to python 3.7 and using spacy 2.1.0

[jsalbr]

Hi, it seems to work fine with spaCy 2.3.0 without building spaCy from source.

# installed spacy 2.3.0 then
git clone https://github.com/huggingface/neuralcoref.git
cd neuralcoref; pip install -r requirements.txt; pip install -e .

But I had to change the requirements.txt to make it work, because here it says spacy>=2.1.0,<2.2.0

Could you please update the requirements.txt

I'd like to use neuralcoref in a book and therefore easy setup would be great!

[bodak]

The approach that works for us in a CI setting is installing spacy, cython and numpy strictly before neuralcoref. Afterwards you can just pip install directly from Github (i.e. pip handles the cloning to tmp).

Example Makefile:

...
build:
	pip3 install numpy==1.19.0 cython==0.29.20 spacy==2.3.0
        pip3 install git+https://github.com/huggingface/neuralcoref.git
...

Note if you have it part of larger NLP stack, you can also add git+https://github.com/huggingface/neuralcoref.git to requirements.txt and pip3 install -r requirements.txt.

[svlandeg]

Hi, it seems to work fine with spaCy 2.3.0 without building spaCy from source.

# installed spacy 2.3.0 then
git clone https://github.com/huggingface/neuralcoref.git
cd neuralcoref; pip install -r requirements.txt; pip install -e .

But I had to change the requirements.txt to make it work, because here it says spacy>=2.1.0,<2.2.0

Could you please update the requirements.txt

The requirements are now updated. There was a bug in spaCy 2.2 which meant that the pip-installed version of spaCy did not distribute the pxd files which neuralcoref builds against. But this is now fixed in newer versions of spaCy, so you don't have to build spaCy from source anymore.

Thanks for the confirmation!

[morrisalp]

The requirements are now updated. There was a bug in spaCy 2.2 which meant that the pip-installed version of spaCy did not distribute the pxd files which neuralcoref builds against. But this is now fixed in newer versions of spaCy, so you don't have to build spaCy from source anymore.

I just got a segfault when trying to use neuralcoref with spacy 2.3.2 and had to downgrade to spacy 2.1.3 for it to work, even though the requirements file allows spacy>=2.1.0,<3.0.0.

[svlandeg]

Hm, that's odd. Did you build neuralcoref from source after installing spaCy 2.3.2 ?

[morrisalp]

I installed neuralcoref via pip after installing spaCy 2.3.2.

[svlandeg]

Yea, that explains the segfault. You'll need to build neuralcoref from source if you want to use spacy > 2.1.

[fhamborg]

thanks for the great tool. though, im wondering, is neuralcoref actually under active development or what's the overall status? for example, this issue is almost 2 years old. further, in the dev branch the latest commit is 2 months old (and it's only regarding some typos). also, spacy 3.0 has been out for a while and still there's no support.

[svlandeg]

Hi @fhamborg! Since my original post in September 2019, quite a bit has changed. We initially intended to bring this code base up-to-date with the latest spaCy, but then we ended up revamping the internals of spaCy quite a bit for its 3.0 release.

Long story short, at Explosion we're working on a built-in coreference module that will be part of the core spaCy library. You can follow its development here: https://github.com/explosion/spaCy/pull/7264. You'll see that we've been actively working on this. Most of the code is written, but it needs some more debugging and fine-tuning.

I can't speak for HuggingFace and what they intend to do with neuralcoref going forward, though either way this library will always be useful for older versions of spaCy :-)

[fhamborg]

That's great to hear, thank you a lot for the update :)