unsloth
unsloth copied to clipboard
unslothai
Notebook to show vLLM inference
It would have been nice if there were some inference examples like vLLM. So that we know how exactly model inferencing is working.
Inference is the same in vllm as any other model, they have examples in their documentation https://docs.vllm.ai/en/latest/getting_started/quickstart.html
Yep vLLM has some cool docs - would adding it into an Unsloth notebook be helpful?
Yep vLLM has some cool docs - would adding it into an Unsloth notebook be helpful?
Yes, it would be helpful. Actually I was trying to solve classification tasks which would churn out logits and was experimenting with this notebook https://github.com/timothelaborie/text_classification_scripts/blob/main/unsloth_classification.ipynb which wasn't working and I had to do a fast inferencing vLLM. So that is where I am stuck.
Would be glad if some examples are added for vLLM and classification tasks.
It would have been nice if there were some inference examples like vLLM. So that we know how exactly model inferencing is working.
title and text don't match. Title says a vLLM example, not an unsloth example. Please keep question and text consistent or it's relaly hard to go through the git issues
Hi @danielhanchen and @rolandtannous,
I am new to Unsloth and looking to contribute. I came across this issue and wanted to check if it’s still open. If so, I’d be happy to take it on. Please let me know if you’re still looking for someone to address it.
Thanks!
Ok will try add some vLLM examples!
@shimmyshimmer , I can add some vllm and classification task examples. Can I take this issue?
These have already been added if i am not mistaken
checkout the notebooks github repo : https://github.com/unslothai/notebooks
@rolandtannous , You are talking about these, right? https://github.com/unslothai/notebooks?tab=readme-ov-file#vision-multimodal-notebooks