spaCy icon indicating copy to clipboard operation
spaCy copied to clipboard

Add dependency check to project step runs

Open rmitsch opened this issue 3 years ago • 5 comments

Goal

Ensure dependencies specified in requirements.txt are installed. Notes:

  • No tests have been added yet. Dedicated tests would be nice, but we'd need to mess with the test venv for that - not sure that's worth it?
  • Documentation will be added once the code review is through.

Description

Uses pkg_resources to check whether all dependencies can be imported without issues. Note that this doesn't check the complete dependency tree - theoretically a sub-dependency might be missing or mismatched without this check noticing.

Types of change

New feature.

Checklist

  • [x] I confirm that I have the right to submit this contribution under the project's MIT license.
  • [x] I ran the tests, and all new and existing tests passed.
  • [ ] My changes don't require a change to the documentation, or if they do, I've added all required information.

rmitsch avatar Jul 27 '22 13:07 rmitsch

Have you looked at the timing for this yet?

adrianeboyd avatar Jul 27 '22 14:07 adrianeboyd

Have you looked at the timing for this yet?

Not yet, wanted to confirm this approach is acceptable before.

rmitsch avatar Jul 27 '22 14:07 rmitsch

If the basic pkg_resources checks are extremely slow we can immediately stop working on it. (I don't think they will be, though.)

adrianeboyd avatar Jul 27 '22 14:07 adrianeboyd

If the basic pkg_resources checks are extremely slow we can immediately stop working on it. (I don't think they will be, though.)

So far they felt pretty instantaneous. Will benchmark with more extensive requirement files tomorrow.

rmitsch avatar Jul 27 '22 15:07 rmitsch

Running the validation with this cobbled together list of requirements

scipy
umap
transformers>=3.4.0,<4.3.0
spacy-transformers>=1.0.1,<1.1.0
stanza>=1.3.0,<1.5.0
flair>=0.6.0,<1.0.0
ufal.udpipe
spacy-lookups-data>=1.0.0,<1.1.0
ml_datasets==0.2.0a0
spacy-experimental==0.2.0
typer
numpy
tqdm
wasabi
pydantic>=1.0.0,<2.0.0
fastapi>=0.61.1,<0.62.0
aiofiles
uvicorn>=0.11.6,<0.12.0
spacy-huggingface_hub
plotext
spacy_ray
spacy-streamlit>=1.0.0a0
streamlit
wandb>=0.12.4,<0.20.0
floret>=0.10.1,<0.11.0
natto-py
spacy-streamlit>=1.0.0a0
spacy-lookups-data>=1.0.0,<1.1.0
google-cloud-storage
streamlit
presidio-analyzer==2.2.1
presidio-anonymizer==2.2.1
skweak==0.2.13
hmmlearn==0.2.6
nlpaug

takes - on my local machine - 0.33 seconds in an environment which has all of these dependencies installed and 0.21 seconds in one that's missing most of them. I think that's reasonable.

rmitsch avatar Jul 28 '22 08:07 rmitsch

I think the conflicting dependencies message needs more info.

Displaying the full message from VersionConflict.report() now. This should show streamlit 1.9.2 has requirement click<8.1,>=7.0, but you have click 8.1.3. in the warning message. Can you verify that?

rmitsch avatar Aug 16 '22 08:08 rmitsch

The merge undid some other edits?

adrianeboyd avatar Aug 16 '22 08:08 adrianeboyd

The merge undid some other edits?

Weird. But only https://github.com/explosion/spaCy/pull/11226/commits/f58195569c93249d5a1ccbb5fa000706322498e4, right? Re-added the change.

rmitsch avatar Aug 16 '22 08:08 rmitsch