GPULlama3.java icon indicating copy to clipboard operation
GPULlama3.java copied to clipboard

Improve error handling on out of memory exception to provide relevant solution for the llama

Open mikepapadim opened this issue 6 months ago • 1 comments

Currently, if one tries to run a model that requires more than the default memory allocation (e.,g, 7GB), GPULlama3.java throws a tornado out of memory exception.

The error message refers to to a solution to increase GPU heap size through a Tornado flag, but the llama repo expects wrapper calls as --gpu-memory XGB

Solution:

Lets try to catch the exception in TornadoMasterPlan class and throw the appripate message to increase heap size that is relevant to the llama implementation.

mikepapadim avatar Jun 19 '25 18:06 mikepapadim

Hey @mikepapadim , would like to take this issue

dhruvarayasam avatar Jul 20 '25 01:07 dhruvarayasam