Dongfu Jiang

> Hello @jdf-prog! Just to confirm, you were able to launch the server, but only this particular image ran into an issue, correct? Yes, only this particular image. The following...

And the code it will work again after the resize: ```python from vllm import LLM from vllm.sampling_params import SamplingParams from PIL import Image from io import BytesIO import base64 import...

> > > In the mean time, can you modify your model initialization similar to what's in `examples/offline_inference_pixtral.py`? > > ```python > model_name = "mistralai/Pixtral-12B-2409" > max_img_per_msg = 5 >...

Can you post what errors you have encountered during the loading?

```python import os os.environ["CUDA_VISIBLE_DEVICES"] = "0" import llm_blender blender = llm_blender.Blender() # Load Ranker blender.loadranker("llm-blender/PairRM") # load ranker checkpoint ``` For me, the above code, and the codes in the...

About the local files, of course it's possible. simply replace the model_name to your local checkpoint path.

Also encounter this problem, triton==2.2.0 solves it. But it seems there are some version mismatch between torch. Don't know if that will cause some problems.