dstackai
[Examples] Add vLLM example
This issue is stale because it has been open for 30 days with no activity.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Curious to know how vllm is different from this issue?
@bikash119
We'd like to have it as a separate example under the Deployment category (in addition to other categories such as Fine-tuning, Accelerators, etc.).
The example should feature most essential information around using vLLM with dstack.
Note, dstack allows to use vllm with both tasks and services. Should we show both, also explaining briefly the difference?
Apologies for being a noob here. Under Deployment category means, to provide vllm as a serverless inference service on dstack?
Apologies for being a noob here. Under
Deploymentcategory means, to provide vllm as a serverless inference service on dstack?
@bikash119 Currently, there are three categories on https://dstack.ai/examples/: Fine-tuning, Accelerators, and LLMs. Let's add Deployment - can be the first in the list.
Ok, so under Deployment , we should have 2 examples shown
- How to use vLLM for
service - How to use vLLM for
taskDoes this sound ok?
Ok, so under
Deployment, we should have 2 examples shown
- How to use vLLM for
service- How to use vLLM for
taskDoes this sound ok?
Under Deployment, we are going to have one card "vLLM".
This will lead to https://dstack.ai/examples/deployment/vllm.
The sourcecode must be in examples/examples/deployment/vllm/README.md (it's copied to the docs when docs are built). See other cards on https://dstack.ai/examples/ to follow the structure and style.
Let me know if this helps
@peterschmidt85 : I have made the required changes. Verifed by executing mkdocs serve
This issue is stale because it has been open for 30 days with no activity.
This issue is stale because it has been open for 30 days with no activity.
This issue was closed because it has been inactive for 14 days since being marked as stale. Please reopen the issue if it is still relevant.