torchchat icon indicating copy to clipboard operation
torchchat copied to clipboard

[Llava][multimodal] enable Llava in torchchat

Open Gasoonjia opened this issue 1 year ago • 1 comments

This PR aims to enable Llava in torchchat.

TODOs:

  • [x] Create Model and ModelArgs as model definition entrances
  • [x] Support model definition with multiple transformers
  • [ ] Add transformer fusion types
  • [ ] Enable Llava definition
  • [ ] Add model config with checkpoint
  • [ ] Update inference pipeline

Gasoonjia avatar Jul 30 '24 03:07 Gasoonjia

:link: Helpful Links

:test_tube: See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchchat/966

Note: Links to docs will display an error until the docs builds have been completed.

:white_check_mark: No Failures

As of commit 11113ff322b63327f0d19f6ab84bfb9ce29c9119 with merge base 9dc9effd674463b572125d01760cb78752e1912a (image): :green_heart: Looks good so far! There are no failures yet. :green_heart:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

pytorch-bot[bot] avatar Jul 30 '24 03:07 pytorch-bot[bot]