text-generator.io icon indicating copy to clipboard operation
text-generator.io copied to clipboard

Add optional vLLM integration

Open lee101 opened this issue 6 months ago • 0 comments

Summary

  • implement vllm_inference helper
  • integrate vLLM into server with USE_VLLM toggle
  • document vLLM installation and env var
  • provide CLI helper and tests for new logic

Testing

  • pytest -q tests/unit/test_vllm_env.py tests/unit/test_vllm_inference.py
  • ruff check . (fails: Found 297 errors)

https://chatgpt.com/codex/tasks/task_e_6847966145208333bcd0c84d4eb29606

lee101 avatar Jun 10 '25 02:06 lee101