exo icon indicating copy to clipboard operation
exo copied to clipboard
Published 13 hours ago • verified publisher icon exo-explore

Llama 3.3 redownloading in every new session and few other problems

Open VistritPandey opened this issue 1 year ago • 5 comments

For reference, I am using 2 Mac Mini with M4 Pro and 48GB RAM each. Both are connected with Thunderbolt 5 cable. However, I am facing multiple issues:

  1. Llama 3.3-70b is working at 4~5 tokens/sec which is the same number of tokens or less I am getting when using only 1 machine and running the model with Ollama.
  2. After pulling the latest version of the repository, or whenever I restart the machines, it redownloads some of the models
  3. Llama 3.1-8b does not work at all
  4. Getting gRPC error in the logs Screenshot 2024-12-19 at 10 00 12
  5. It is not able to connect to Hugging Face to download the model, which it was able to do before I installed the latest version of MacOS (15.2 (24C101)). If it has something to do with the access token to hugging face, it is already there, and I can use it for other tasks Screenshot 2024-12-19 at 10 05 01
  6. It is stuck at a number beyond which it is not able to download the Llama3.3 that it downloaded before. Maybe it's corrupted somehow?

VistritPandey avatar Dec 19 '24 16:12 VistritPandey

My guess is that's not enough RAM. I have the same problem. https://www.substratus.ai/blog/calculating-gpu-memory-for-llm I'm still trying to figure out how to benchmark this system. when using llama 3.2 1B adding a node 32GB M2 Mac Mini to a single 64GB M4 mac mini varies performance up or down about 10%.

AFDudley avatar Dec 28 '24 22:12 AFDudley

Ohh that maybe a good answer. Thanks! I was under the assumption that it takes all the RAM by default to be considered by the GPU

VistritPandey avatar Dec 29 '24 16:12 VistritPandey

Re: It is not able to connect to Hugging Face to download the model

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)

I ran

/Applications/Python\ 3.12/Install\ Certificates.command and pip install --upgrade certifi

and it seemed to fix it

rolliver avatar Jan 07 '25 21:01 rolliver

Re: It is not able to connect to Hugging Face to download the model

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)

I ran

/Applications/Python\ 3.12/Install\ Certificates.command and pip install --upgrade certifi

and it seemed to fix it

This actually fixed the issue for me! Thank you!

Although when I restart my machine, I had to re-run it

VistritPandey avatar Jan 07 '25 22:01 VistritPandey

Re: It is not able to connect to Hugging Face to download the model

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)

I ran

/Applications/Python\ 3.12/Install\ Certificates.command and pip install --upgrade certifi

and it seemed to fix it

Thank you @rolliver

Fixed

ThaddeusJiang avatar Jan 17 '25 10:01 ThaddeusJiang