text-generator.io icon indicating copy to clipboard operation
text-generator.io copied to clipboard

Switch to vLLM with custom stopping criteria

Open lee101 opened this issue 7 months ago • 0 comments

Summary

  • add vLLM based inference helper with min-probability and sentence stopping
  • integrate vLLM into the FastAPI server when available
  • make tests/conftest resilient if httpx is missing
  • add basic unit test for vLLM inference

Testing

  • pytest -q (fails: ModuleNotFoundError: No module named 'cachetools')

https://chatgpt.com/codex/tasks/task_e_683fe5b06bf083338fb1ba3540b415dc

lee101 avatar Jun 04 '25 07:06 lee101