Feature: Magma Integration with Ollama and vLLM.

[Praveenstein]

Issue Description:

I'd like to request a feature that integrates the Magma foundation model with Ollama and vLLM. This would significantly enhance the accessibility and performance of Magma for multimodal AI agent development.

Rationale:

Magma, as a foundational multimodal AI agent model, boasts strong capabilities in perceiving the multimodal grounded world and executing goal-driven actions. Integrating it with Ollama and vLLM would provide several key benefits:

• Ollama Integration:
  • Ollama simplifies the process of running large language models locally. By integrating Magma with Ollama, users could easily download, manage, and run the model on their own hardware, fostering local development and experimentation.
  • This would democratize access to Magma, allowing developers and researchers to work with it without relying on cloud-based services.
• vLLM Integration:
  • vLLM optimizes LLM inference and serving, resulting in significant performance improvements. By integrating Magma with vLLM, users could achieve higher throughput and lower latency when running the model, particularly for real-time applications.
  • This will allow for the fast and efficient serving of a multimodal agent.
• Enhanced Multimodal Agent Development:
  • Combining Magma's multimodal capabilities with Ollama's ease of use and vLLM's performance would create a powerful and accessible platform for developing advanced multimodal AI agents.
  • This integration would provide a streamlined workflow for developers, from model deployment to inference.

Proposed Implementation:

• Provide an Ollama modelfile for Magma, allowing users to easily download and run the model using Ollama.
• Develop vLLM support for Magma, ensuring compatibility with its architecture and multimodal inputs.
• Provide clear documentation and examples demonstrating how to use Magma with Ollama and vLLM.

Benefits:

• Increased accessibility to Magma for local development and experimentation.
• Improved performance and efficiency for Magma inference.
• Streamlined workflow for multimodal AI agent development.
• Fosters innovation and research in multimodal AI.

Thank you for considering this feature request.

[jwyang]

Hi, @Praveenstein , thanks for your suggestion! I do not have any experience to integrate into Ollama or vLLM. Not sure how much effort to spend. Any idea?

[llv22]

@jwyang maybe you can request model like https://github.com/vllm-project/vllm/issues/14321 for magma. They can manage to add to model layer https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models

[jwyang]

thanks for sharing the example! @llv22

[Umutayb]

Any progress on this? I was hoping to consume a more streamlined api via ollama