Qingye Meng

icon indicating an email [email protected]

icon company ColorfulClouds Tech. Co. Ltd. icon location Beijing icon location I am interested in LLMs, interpretability of AI.

Results 1 comments of Qingye Meng

MFU drops significantly when using megablox with more experts

> Thanks for reaching out! We did some internal benchmarks about DeepSeek v3 and Llama4 Maverick on [Cloud v5p](https://cloud.google.com/tpu/docs/v5p), using megablox, adamw, dtype=bf16, weight_dtype=f32, and FSDP sharding. The performance is...