torchchat icon indicating copy to clipboard operation
torchchat copied to clipboard

[Llava][multimodal] enable Llava in torchchat

Open Gasoonjia opened this issue 6 months ago • 1 comments

This PR aims to enable Llava in torchchat.

TODOs:

  • [x] Create Model and ModelArgs as model definition entrances
  • [x] Support model definition with multiple transformers
  • [ ] Add transformer fusion types
  • [ ] Enable Llava definition
  • [ ] Add model config with checkpoint
  • [ ] Update inference pipeline

Gasoonjia avatar Jul 30 '24 03:07 Gasoonjia