Cannot install OpenVINO tokenizer 2024.4 with optimum-intel as given in readme

[dhandhalyabhavik]

Describe the bug According to this readme, I am supposed to install openvino tokenizer and other libraries using these commands,

export PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cpu https://storage.openvinotoolkit.org/simple/wheels/pre-release"
pip3 install --pre "optimum-intel[nncf,openvino]"@git+https://github.com/huggingface/optimum-intel.git  openvino_tokenizers==2024.4.* openvino==2024.4.*

But facing this error,

ERROR: Cannot install openvino-tokenizers==2024.4.0.0rc1, openvino==2024.4.* and optimum-intel[nncf,openvino]==1.20.0.dev0+f1517e3 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested openvino==2024.4.*
    openvino-tokenizers 2024.4.0.0rc1 depends on openvino~=2024.4.0.0.dev
    optimum-intel[nncf,openvino] 1.20.0.dev0+f1517e3 depends on openvino<2024.4 and >=2023.3; extra == "openvino"

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

To Reproduce Just follow above steps.

Expected behavior tokenizer flag support --utf8_replace_mode replace for recent tokenizer library.

Logs

Configuration

• openvino tokenizer doesn't even have --version flag to check version. Don't know what I am suppose to write here.

Additional context

[mzegla]

Seems like new limit on OV version has been added to optimum-intel: https://github.com/huggingface/optimum-intel/commit/cff990218b93313519b15f15bf1005a90ca8aeab I'll try to find out why it was added and when we could lift that limitation. Meanwhile you can try to use optimum-intel commit before that change.