Compatibility Issue when using v0.2.0 with transformers and tensorflow

[aws-tianquaw]

I recently encountered some compatibility issue when using sentencepiece v0.2.0 together with latest transformers and tensorflow packages. When I ran some Python script that imports AutoProcessor class from transformers, the Python kernel was killed with the following error message:

ERROR: Flag 'minloglevel' was defined more than once (in files 'src/error.cc' and 'home/conda/feedstock_root/build_artifacts/abseil-split_1720857154496/work/absl/log/flags.cc').

I tried to downgrade sentencepiece to v0.1.99 and then the issue was solved.

Commands to reproduce the issue:

conda create -n "py311" python=3.11
source activate py311
conda install -c conda-forge transformers==4.45 tensorflow==2.17 sentencepiece==0.2
python -c "from transformers import AutoProcessor"

[Genjidegaulle]

Running into a similar issue, with the same temporary resolution of downgrading sentencepiece to 0.1.99.

[aws-tianquaw]

Hi team, any update on this issue?

[odusseys]

bumping this - it is impossible to use transformers and 0.2.0 together - e.g. mistral-common requires 0.2.0 but we need to run both together without downgrading to 0.1.99

[reneleonhardt]

This project doesn't look maintained anymore, I doubt it will be fixed. AI dependencies are being released every day 🤷

[taku910]

This occurs when sentencepiece is built with an external Abseil package (_USE_EXTERNAL_ABSL is enabled), and another library attempts to define minloglevel using the same Abseil library.

This is not a pip package that we distribute (_USE_EXTERNAL_ABSL is disabled), so it is essentially unsupported. We believe conda has created its own package, so please contact the author of that package. Since we know the cause, we conclude that this is an issue with the build environment.

Thank you.