LLaMA-Factory icon indicating copy to clipboard operation
LLaMA-Factory copied to clipboard

Published 20 hours ago • verified publisher icon hiyouga

llama3.1 405B fp8 support

Open mces89 opened this issue 7 months ago • 0 comments

Reminder

  • [X] I have read the README and searched the existing issues.

System Info

n/a

Reproduction

n/a

Expected behavior

No response

Others

Hi, I see the latest master has supported the 405B, but it needs at least 2 nodes to do the inference, how do we do that? Also Is it possible to support 405B fp8 so it can be run within 1 A100 node?

mces89 avatar Jul 26 '24 22:07 mces89