LongVILA - Base LLM not declared in the Paper?

[liyucheng09]

I'd like to ask the base LLM of the following LongVILA checkpoint:

• Efficient-Large-Model/Llama-3-LongVILA-8B-128Frames
• Efficient-Large-Model/Llama-3-LongVILA-8B-256Frames
• Efficient-Large-Model/Llama-3-LongVILA-8B-512Frames

This was named with Llama-3, however, as quote from the paper:

It is unclear what's the exact base LLM of LongVILA.

[Lyken17]

LongVILA was originally trained with LLama3 (so as the releaesd models) and recently updated to Qwen2 backbone. @yukang2017 can help confirm

[liyucheng09]

@Lyken17 thank you Ligeng for the prompt reply.

@yukang2017 would love to have the Qwen2 version on huggingface, so we can reproduce the results in the submission. thanks!

[yukang2017]

Thanks for your interests in our work. We are waiting for legal permission from NVIDIA to release qwen2 models. It should be approved this week. I will let you know here when it is release on hf. Thanks for your patience.

[yukang2017]

@liyucheng09 Hi, our code and models have been released. You can find it below. The benchmark comparison uses Efficient-Large-Model/qwen2-7b-longvila-256f.

Updated results is in the paper.

Paper: https://arxiv.org/pdf/2408.10188 Code: https://github.com/NVlabs/VILA/tree/main/longvila Model: https://huggingface.co/collections/Efficient-Large-Model/longvila-66c3fce79284c8209f119b32

[liyucheng09]

@yukang2017 Hi Yukang, thanks for updating the latest model. Will try it soon. Close as solved.

[liyucheng09]

@yukang2017 Hi Yukang, the repo seems to be largely updated since the NVILA release.

the old command to reproduce the results in the paper is not valid anymore.

Any ideas how to run the experiments in the LongVILA paper?

[liyucheng09]

@yukang2017 @zhijian-liu Hi, it seems this repo is largely broken after the NVILA relase.

The vila-run and vila-eval all based on scripts under scripts/v1_5/, but these are removed in @zhijian-liu's recent NVILA release. Is this a merging issue or there will be a new version of vila-run?

Error see below.

(nvila) (base) [email protected]@GCRAZGDL1694:~/vlm/nvila$ vila-run -m eval -J longvila-7b-256f/lmms-nextqa_mc_test-32 scripts/v1_5/eval/lmms.sh nextqa_mc_test-32 Efficient-Large-Model/LongVILA-7B-256f auto 12
No slurm installed. Ignore slurm-related args.
scripts/v1_5/eval/lmms.sh nextqa_mc_test-32 Efficient-Large-Model/LongVILA-7B-256f auto 12
/bin/sh: 1: scripts/v1_5/eval/lmms.sh: not found
returncode: 127
Job finished with exit code 127

(nvila) (base) [email protected]@GCRAZGDL1694:~/vlm/nvila$ vila-eval -m Efficient-Large-Model/LongVILA-7B-256f -c auto -nf 128 -t vnbench_val
2024-12-20 19:55:00.621 | INFO     | llava.cli.eval:main:70 - Running evaluation for 'longvila-7b-256f' on 1 tasks: ['vnbench_val-128']
2024-12-20 19:55:00.621 | INFO     | llava.cli.eval:main:124 - Running 'vila-run -m eval -J longvila-7b-256f/vnbench_val-128 scripts/v1_5/eval/vnbench.sh val-128 Efficient-Large-Model/LongVILA-7B-256f auto 128 12'
2024-12-20 19:55:02.624 | ERROR    | llava.cli.eval:main:151 - Error running 'vnbench_val-128' evaluation (return code: 127)
2024-12-20 19:55:02.625 | INFO     | llava.cli.eval:main:188 - Saved all metrics to 'runs/eval/longvila-7b-256f/metrics.json'
2024-12-20 19:55:02.626 | INFO     | llava.cli.eval:main:194 - Results:
┌──────────┬─────────┐
│ Metric   │ Value   │
├──────────┼─────────┤
└──────────┴─────────┘

[yukang2017]

Hi @liyucheng09 ,

Sorry for the bug. I fix it in this pr. You can try this repo, https://github.com/yukang2017/VILA/tree/main, before it was merged.

https://github.com/NVlabs/VILA/pull/170

Regards, Yukang Chen

[liyucheng09]

@yukang2017 many thanks, will try it very soon.

[liyucheng09]

@yukang2017 Hi Yukang, can you help to confirm the conv_template used for Efficient-Large-Model/qwen2-7b-longvila-256f?

conv_templates = {
    "auto": conv_auto,
    "hermes-2": hermes_2,
    "llama_3": llama_3_chat,
    "v1": conv_vicuna_v1,
    "vicuna_v1": conv_vicuna_v1,
    "plain": conv_llava_plain,
}

I cannot reproduce the results in the paper, and I suspect it's due to a wrong conv template.

[liyucheng09]

@Lyken17 Hi Yukang, one more small question on reproduction. Should I specify add_newline_token in v-niah test?

[yukang2017]

Hi @liyucheng09 , I think we do not need to set conv_template when use vila-run evaluation. It can find the conv template from the tokenizer config json.

I did not set add_newline_token before. What are the results do yo get?

Did you follow the instructions here for evaluation? https://github.com/NVlabs/VILA/tree/main/longvila#evaluations