Kiv Chen
Kiv Chen
Generation Model Test & MMMU results . @zhaochenyang20 @yhyang201 ### MMMU `pixtral-12b` scored around 44 with _direct_ prompt; but Mistral-AI claimed that its MMMU score goes up to 50.9 with...
multimodal part just aligned
> I think that you should use the official script [MMMU-Benchmark/MMMU](https://github.com/MMMU-Benchmark/MMMU) On it, but I think official impl also randomly selects answers if parsing fails. https://github.com/MMMU-Benchmark/MMMU/blob/7b38a4ff9358f15e3b22394e75052297f289a474/mmmu/utils/eval_utils.py#L39
> > I think that you should use the official script [MMMU-Benchmark/MMMU](https://github.com/MMMU-Benchmark/MMMU) > > On it, but I think official impl also randomly selects answers if parsing fails. [MMMU-Benchmark/MMMU@`7b38a4f`/mmmu/utils/eval_utils.py#L39](https://github.com/MMMU-Benchmark/MMMU/blob/7b38a4ff9358f15e3b22394e75052297f289a474/mmmu/utils/eval_utils.py#L39) It...
Looks like chunked prefill for VLM is tougher than expected. Was about to look into that.
I might be able to test out #5250 specifically on llama4-scout. However, my available hw has limited vram per node. Does sglang currently support multi-node EP for llama4? Like 3nodes...
As PR statement is updated: (Reverted) - Multimodal is enabled by default for registered mm models. `--disable-multimodal` is added for models like llama4. - now chunked prefill is enabled for...
M3.1-S looks solid for my use case — mainly need an mmlm that runs on low-spec hw. That said, I'd mention a few notes: - it's basically a Mistral text...
Found unusual impl while testing vllm for pixtral: [starting from pixtral support#8377](https://github.com/vllm-project/vllm/pull/8377/files#diff-dc4afb08b7a54bb237ee02180225d41d542166342c1f09d41978b7893f332dd0), Mistral-AI configs need separate handling, as their newly published models _no longer_ complies with HF standards. SGL appears...
> > Found unusual impl while testing vllm for pixtral: [starting from pixtral support#8377](https://github.com/vllm-project/vllm/pull/8377/files#diff-dc4afb08b7a54bb237ee02180225d41d542166342c1f09d41978b7893f332dd0), Mistral-AI configs need separate handling, as their newly published models _no longer_ complies with HF standards....