Cyrus Leung

[Core] Support image processor

1

I have implemented a plugin architecture (`MultiModelPlugin`) over `MultiModalData` to define how each modality type should be preprocessed before being passed to the model as keyword arguments. This preserves the...

[CI/Build] Move `test_utils.py` to `tests/utils.py`

2

Since #4335 was merged, I've noticed that the definition of `ServerRunner` in the tests is the same as in the test for OpenAI API. I have moved the class to...

[Frontend] Refactor prompt parsing

9

I have refactored the prompt parsing logic by moving it to the base class `OpenAIServing`, so that it can be shared between the Chat Completions, Completions and Embeddings APIs. The...

[Core] Consolidate prompt arguments to LLM engines

4

Currently, `LLM.generate` (and similar methods in `LLMEngine` and `AsyncLLMEngine`) accept `prompt`, `prompt_token_ids` and `multi_modal_data` separately. This PR consolidates them into `PromptInputs` so that only a single argument has to be...

FIX #4617

[Bugfix][Model] Add base class for vision-language models

1

This PR adds a base class `VLMBase` to avoid importing `LlavaForConditionalGeneration` in `vllm/model_executor/model_loader/loader.py`, thus solving #4807. Along the way, I have also ported the improved error handling logic regarding `image_feature_size`...

[Doc] Add page for `PoolingParams`

1

This PR builds on #4710 by adding a page for `PoolingParams` which was recently introduced by #3734.

[Draft][CI/Build] Optimize models tests

1

The models tests keep getting interrupted (presumably due to running too long). This PR attempts to reduce the running time via: - Downloading all models at the same time when...

[Frontend] Support GPT-4V Chat Completions API

5

I have refactored the OpenAI-compatible server to support image input in [OpenAI Chat Completions API](https://platform.openai.com/docs/api-reference/chat/create), as specified by [GPT-4 with Vision (GPT-4V)](https://platform.openai.com/docs/guides/vision). This enables easy access to VLMs in vLLM...

[Core] Omni-Modal Embedding, Vector Index and Retriever

2

# Description This PR lays the groundwork for extending the multi-modal support to other modalities (such as audio). The main components of this PR are: - [`Modality`](https://github.com/DarkLight1337/llama_index/blob/omni-modal/llama-index-core/llama_index/core/embeddings/omni_modal_base.py#L171): Encapsulates the embedding-agnostic...