vision language model support

[7uk3y]

trafficstars

Feature request

The developments in the robotics community around RT-2 show a lot of potential for VLMs but the hardware constraints for small developers makes it difficult to deploy RT-2 level performance with off the shelf hardware suitable for robotics. A potential path forward would be using very small models that have a library of loras to access when the environment calls for them. Lorax looks like it might be a way forward for smaller devs and robot enthusiasts to use and experiment with VLM architectures

Motivation

robots as a hobby

Your contribution

testing and refining available models

[tgaddair]

Thanks for the filing this issue! I think this shouldn't be too hard to add support for some popular ones like LLaVA.

[pbarker]

Hey @tgaddair I would be up to give this a try, could you just give me a basic rundown of what you think needs to be done here to save me some head scratching