bug: huggingface-hub 1.0.0 release breaks existing evaluate releases

[nvluxiaoz]

We encountered the following issues today with pip3 install evaluate==0.4.5 rouge_score==0.1.2. The issue is

    token = hf_api.HfFolder.get_token()
            ^^^^^^^^^^^^^^^
AttributeError: module 'huggingface_hub.hf_api' has no attribute 'HfFolder'

The script is simple:

import evaluate


def calculate_rouge_score(predictions, references):
    """
    Compute Rouge score between predictions and references.
    Args:
        predictions: List of predictions.
        references: List of references.
    Returns:
        Rouge score. Format: {
            "rouge1": float,
            "rouge2": float,
            "rougeL": float,
            "rougeLsum": float
        }
    """
    rouge = evaluate.load("rouge")
    return rouge.compute(predictions=predictions, references=references)

This issue can be resolved after we fix huggingface-hub==0.36.0. It is due to https://github.com/huggingface/evaluate/blob/v0.4.5/setup.py#L54 does not check major version < 1.0.0 and pip uses latest available version. This affects previous stable versions. Could you enable stricter dependency checks?

[sfc-gh-sbekman]

And the suggested workaround doesn't work in HF Transformers @ main

$ uv pip install huggingface-hub==0.36.0
Using Python 3.12.11 environment at: /home/yak/miniconda3/envs/dev
Resolved 13 packages in 5ms
Uninstalled 1 package in 3ms
Installed 1 package in 13ms
 - huggingface-hub==1.0.0rc6
 + huggingface-hub==0.36.0
 
$ RUN_SLOW=1 pytest tests/deepspeed
ImportError while loading conftest '/code/users/stas/github/transformers-alst-integration/conftest.py'.
conftest.py:27: in <module>
    from transformers.testing_utils import (
src/transformers/__init__.py:27: in <module>
    from . import dependency_versions_check
src/transformers/dependency_versions_check.py:57: in <module>
    require_version_core(deps[pkg])
src/transformers/utils/versions.py:116: in require_version_core
    return require_version(requirement, hint)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/transformers/utils/versions.py:110: in require_version
    _compare_versions(op, got_ver, want_ver, requirement, pkg, hint)
src/transformers/utils/versions.py:43: in _compare_versions
    raise ImportError(
E   ImportError: huggingface-hub==1.0.0.rc6 is required for a normal functioning of this module, but found huggingface-hub==0.36.0.
E   Try: `pip install transformers -U` or `pip install -e '.[dev]'` if you're working with git main

[sfc-gh-sbekman]

I tested the problem has been resolved in evaluate@main - please kindly make a new release asap! Thank you!

[sfc-gh-sbekman]

correction: evaluate==0.4.6 works just fine, I'm looking at which package forced evaluate==0.4.5 which is the culprit of this issue.

[sfc-gh-sbekman]

Perhaps it's transformers that need to check it's at least evaluate==0.4.6 - let me make a PR https://github.com/huggingface/transformers/pull/41920

[nvluxiaoz]

by the way evaluate==0.4.5 is forced by my own software. It worked all fine previously but broke yesterday.

[stas00]

switch to evaluate==0.4.6 and it should work fine.

[stas00]

fwiw, my PR got merged in HF Transformers.