NeMo icon indicating copy to clipboard operation
NeMo copied to clipboard

Fix bugs in LLaVA NeXT model with padding and CPU initialization options

Open eagle705 opened this issue 7 months ago • 0 comments

[!IMPORTANT]
The Update branch button must only be pressed in very rare occassions. An outdated branch is never blocking the merge of a PR. Please reach out to the automation team before pressing that button.

What does this PR do ?

This PR addresses bug fixes in the LLaVA NeXT model, specifically adding padding functionality and introducing a use_cpu_init argument to export nemo model to HF format without requiring a GPU or avoiding OOM error

Collection: vlm/llava_next

Changelog

  • Added pad_to_multiple_of to support training without packed sequences.
  • Introduced use_cpu_init argument to enable HFExporter functionality without requiring a GPU.

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this 

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR. To re-run CI remove and add the label again. To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

  • [ ] Make sure you read and followed Contributor guidelines
  • [ ] Did you write any new necessary tests?
  • [ ] Did you add or update any necessary documentation?
  • [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • [ ] Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

  • [ ] New Feature
  • [x] Bugfix
  • [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

@abhinavg4

Anyone in the community is free to review the PR once the checks have passed. Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

  • Related to # (issue)

eagle705 avatar May 13 '25 13:05 eagle705