support new processor arg video_maxlen_ttl
What does this PR do?
Fixes #7693 Replace #7700 (too many changes in it to check, so open this PR with a new branch)
Before submitting
- [x] Did you read the contributor guideline?
- [ ] Did you write any new necessary tests?
When I run HF_ENDPOINT=https://hf-mirror.com make test, I encountered these errors:
========================================================================================================= short test summary info ==========================================================================================================
FAILED tests/data/processor/test_unsupervised.py::test_unsupervised_data[16] - ValueError: PPO only accepts wandb or tensorboard logger.
FAILED tests/e2e/test_chat.py::test_chat - AssertionError: assert '/raw' == '_rho'
- _rho
+ /raw
FAILED tests/e2e/test_chat.py::test_stream_chat - AssertionError: assert '़़े़ें' == '_rho'
- _rho
+ ़़े़ें
FAILED tests/model/model_utils/test_add_tokens.py::test_add_tokens[False] - TypeError: ModelArguments.__init__() got an unexpected keyword argument 'add_tokens'
FAILED tests/model/model_utils/test_add_tokens.py::test_add_tokens[True] - TypeError: ModelArguments.__init__() got an unexpected keyword argument 'add_special_tokens'
========================================================================= 5 failed, 107 passed, 10 skipped, 6 xfailed, 1 xpassed, 7 warnings in 351.15s (0:05:51) ==========================================================================
make: *** [Makefile:24: test] Error 1
I still don't know why this would happen. Do you have any ideas? @hiyouga
After updating transformers,
$ pip show transformers
Name: transformers
Version: 4.51.3
Summary: State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow
Home-page: https://github.com/huggingface/transformers
Author: The Hugging Face team (past and future) with the help of all our contributors (https://github.com/huggingface/transformers/graphs/contributors)
Author-email: [email protected]
License: Apache 2.0 License
Location: /home/phd-chen.yirong2/anaconda3/envs/vllm/lib/python3.12/site-packages
Requires: filelock, huggingface-hub, numpy, packaging, pyyaml, regex, requests, safetensors, tokenizers, tqdm
Required-by: compressed-tensors, llamafactory, peft, trl, vllm, xgrammar
these 3 errors still remain:
========================================================================================================= short test summary info ==========================================================================================================
FAILED tests/data/processor/test_unsupervised.py::test_unsupervised_data[16] - ValueError: PPO only accepts wandb or tensorboard logger.
FAILED tests/model/model_utils/test_add_tokens.py::test_add_tokens[False] - TypeError: ModelArguments.__init__() got an unexpected keyword argument 'add_tokens'
FAILED tests/model/model_utils/test_add_tokens.py::test_add_tokens[True] - TypeError: ModelArguments.__init__() got an unexpected keyword argument 'add_special_tokens'
========================================================================== 3 failed, 110 passed, 9 skipped, 5 xfailed, 2 xpassed, 7 warnings in 328.52s (0:05:28) ==========================================================================
make: *** [Makefile:24: test] Error 1
Sorry for not applying pip install -e .[dev].
But 1 error still remains after that:
========================================================================================================= short test summary info ==========================================================================================================
FAILED tests/data/processor/test_unsupervised.py::test_unsupervised_data[16] - ValueError: PPO only accepts wandb or tensorboard logger.
========================================================================== 1 failed, 112 passed, 9 skipped, 5 xfailed, 2 xpassed, 7 warnings in 323.25s (0:05:23) ==========================================================================
make: *** [Makefile:24: test] Error 1
Sorry for not applying
pip install -e .[dev]. But 1 error still remains after that:========================================================================================================= short test summary info ========================================================================================================== FAILED tests/data/processor/test_unsupervised.py::test_unsupervised_data[16] - ValueError: PPO only accepts wandb or tensorboard logger. ========================================================================== 1 failed, 112 passed, 9 skipped, 5 xfailed, 2 xpassed, 7 warnings in 323.25s (0:05:23) ========================================================================== make: *** [Makefile:24: test] Error 1
This error disappears with transformers== 4.49.0.
Now I'm working to add some tests regarding this PR in test_mm_plugin.py.