[Bug]: automatic HF Model Upgrade breaking code

[Mumpitz]

Describe the bug

Recently the flair/ner-multi Model on Hugging Face was updated, which broke the loading with Flair v0.11.3. The Upgrade of the Model happened automatically and broke some production code and even the Hugging Face Inference API has issues with the new Model. I could not see a way to pin the Model Version to v0.8.0, which is working with the older Flair Versions prior to v0.12.x.

To Reproduce

from flair import SequenceTagger
tagger = SequenceTagger.load('ner-multi')

Expected behavior

Tagger loads successfully an be used to predict Senteces or do other Stuff.

Logs and Stack traces

Traceback (most recent call last):
  File "/path/to/.local/share/virtualenvs/Project-6mYypoKI/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3433, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-ddd854651e00>", line 4, in <module>
    tagger = SequenceTagger.load('ner-multi')
  File "/path/to/.local/share/virtualenvs/Project-6mYypoKI/lib/python3.9/site-packages/flair/nn/model.py", line 144, in load
    model = cls._init_model_with_state_dict(state)
  File "/path/to/.local/share/virtualenvs/Project-6mYypoKI/lib/python3.9/site-packages/flair/models/sequence_tagger_model.py", line 624, in _init_model_with_state_dict
    return super()._init_model_with_state_dict(
  File "/path/to/.local/share/virtualenvs/Project-6mYypoKI/lib/python3.9/site-packages/flair/nn/model.py", line 74, in _init_model_with_state_dict
    model = cls(**kwargs)
  File "/path/to/.local/share/virtualenvs/Project-6mYypoKI/lib/python3.9/site-packages/flair/models/sequence_tagger_model.py", line 112, in __init__
    embedding_dim: int = embeddings.embedding_length
AttributeError: 'dict' object has no attribute 'embedding_length'

Screenshots

No response

Additional Context

No response

Environment

Dev System Versions:

Flair

0.11.3

Pytorch

1.13.0

Transformers

4.24.0

GPU

False

Production System Versions:

Flair

0.11.3

Pytorch

1.13.0+cu117

Transformers

4.24.0

GPU

True

[alanakbik]

Hello @Mumpitz this unfortunately is a known issue (see #3187) that was caused by the torch 2.0 update. I fixed it for some models, but that also caused other problems. Since resources are limited, it might take a bit before this is fully fixed.

In the meantime, your best option is to either use an older torch version (anything before 2.0) or use one of the "large" models (like ner-english-large) which can also do at least some multilingual inference. We are also working on newer and better multilingual models, so at some point in the future, these will also be released.

[Mumpitz]

We define a custom cache_root, so we have the older models saved locally. Our current solution is to specify the model files directly out of the cache folder.

Would be nice to have the possibility to check for available Versions and pick a specific one, but checking would need Hugging Face to provide a proper API for this. Did not find something like this so far.

I'd like to be able to load a model and on error fall back to an earlier version that worked and notify me about that.

[alanakbik]

Hm looking at this now, perhaps best would be to revert to the previous model. This will break all torch 2.0 code, but at least existing code would work again.

Could you test whether https://huggingface.co/flair/ner-german is working in your setup? I also updated this model but I think this one should work.

[helpmefindaname]

Hi, I don't think that ner-german will work, as the traceback indicates that the older version (<0.12) doesn't handle the serialization of the new models.

However, @Mumpitz you can load old revisions from the HF-hub, e.g. SequenceTagger.load("flair/ner-multi@b4f9c84fc84d2b1a687bf3f38d15218129e1d202") will load the model that is still compatible with 0.11.

[Mumpitz]

Actually the ner-german Model is part of the project as well, and yes it was causing the same issues after i found the workaround for the ner-multi Model. So i used that Workaround on all loaded Models.

The @-Style is something i wasn't aware of so far. Is this what is specified after the dot in the Filename, or where does it come from?

My current loading looks as follows, where the resources/.flair folder is the new cache_root:

    loaded_taggers['ner-multi'] = MultiTagger.load(
        'resources/.flair/models/ner-multi/a25dcf292992753787b66a693ab5fd5d03444c2b1494fd39c9be6a07d14aa0b7'
        '.b7085be4373491dc725f55b30bea1ba20458e692558435c2f3ea1366277bd8bc'
    )

My new thought would be, to specify it as Multitagger.load('ner-multi@b7085...') to load the older Version for a bit cleaner Solution, or is this a different Hash? Would i need to specify flair/ in front for the model to be found? Oh wait, i guess i need to specify the commit hash found at https://huggingface.co/flair/ner-multi/commits/v0.8.0 for example. Ok makes sense.

[Mumpitz]

So thinking about a loading mechanism, i came up with the idea of using the HfApi model_info() in conjunction with the flair .load(), to prevent the code from breaking.

Before loading a model, check the sha of the model_info against a database or different persistent storage, if there is a new revision. If the sha does not exist, its a newer revision. Try to load it, if it breaks, mark it as breaking in the DB. If there is no new revision, load the last one that was marked as working.

Does this make sense some how? In the error case, i would add a notification behaviour, so that i know there is something going on and i might do adapt something in order to use the most recent versions.

[ngramp]

Thanks for leaving the buggy code up on HF that completely breaks people's venv and still doesn't work. gj