text-generator.io
text-generator.io copied to clipboard
Add optional vLLM integration
Summary
- implement
vllm_inferencehelper - integrate vLLM into server with
USE_VLLMtoggle - document vLLM installation and env var
- provide CLI helper and tests for new logic
Testing
pytest -q tests/unit/test_vllm_env.py tests/unit/test_vllm_inference.pyruff check .(fails: Found 297 errors)
https://chatgpt.com/codex/tasks/task_e_6847966145208333bcd0c84d4eb29606