Ronald Luc

12 B model < 40 GB VRAM => Professional GPU with 40 GB VRAM (P100, A100, H100...) 6.9 B model < 24 GB VRAM => Prosumer GPU with 24 GB...

If you can run other GPT models, should be possible as the underlying architecture is the same as for even the smallest GPT model. I have a friend who struggled...

We've successfully deloyed models on real robots by embedding rl_games in the ROS node inferencing the policy. We had to also modify rl_games directly. It runs well on OrangePi5B cpu....