Getting frequent replicate.exceptions.ModelError on small, text only queries using python API, llama-3-70b-instruct model

[gthaker]

I get this error. If I rerun it usually goes thru fine but dies again down the road. (I am trying 300 total question from a professional exam.) I had no trouble running this with llama-2-70b model, i did it many times.

Traceback (most recent call last):
  File "/export/home/gthaker/repos/radonc_inSvc_LLM/run_llm.py", line 58, in <module>
    key, answer, delta = ask_llama_2(ques)
  File "/export/home/gthaker/repos/radonc_inSvc_LLM/llama_2_api.py", line 35, in ask_llama_2
    output = replicate.run(model,
  File "/export/home/gthaker/repos/radonc_inSvc_LLM/venv/lib/python3.10/site-packages/replicate/client.py", line 157, in run
    return run(self, ref, input, **params)
  File "/export/home/gthaker/repos/radonc_inSvc_LLM/venv/lib/python3.10/site-packages/replicate/run.py", line 61, in run
    raise ModelError(prediction.error)
replicate.exceptions.ModelError

[mattt]

Hi @gthaker. I'm sorry that you're occasionally having issues running meta/meta-llama-3-70b-instruct. Can you share a link to or logs for any failed predictions?

replicate.exceptions.ModelError indicates a problem with the model, so there's nothing to be done with the client itself. What you could try to do is wrap your call in a try/catch and implement retry logic to suit your needs.

[gthaker]

Can I send you the link in a private message? Prob. best not share the query with everyone.

GHT

On Sun, Jun 2, 2024 at 5:49 AM Mattt @.***> wrote:

Hi @gthaker https://github.com/gthaker. I'm sorry that you're occasionally having issues running meta/meta-llama-3-70b-instruct. Can you share a link to or logs for any failed predictions?

replicate.exceptions.ModelError indicates a problem with the model, so there's nothing to be done with the client itself. What you could try to do is wrap your call in a try/catch and implement retry logic to suit your needs.

— Reply to this email directly, view it on GitHub https://github.com/replicate/replicate-python/issues/306#issuecomment-2143777969, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC7QKFDOBCNV7UHGBW5IAJLZFLTEHAVCNFSM6AAAAABIUKYSPWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBTG43TOOJWHE . You are receiving this because you were mentioned.Message ID: @.***>

[Y-Sui]

Have the same issue here. The error still exists not only for meta/meta-llama-3-70b-instruct, but also for model mixtral-8x7b-instruct-v0.1.