API & --notebook are a burn-in test, chat extensions work fine

[StonkGodCapital]

Have read the past issues regarding the API and implemented the various suggestions.

Currently starting the app with --no-stream --listen --model gpt-j-6B

You can hit the API just fine, the prompt is received and printed to console properly however when you get to the generation step the GPU ramps up and just stays pegged indefinitely. Let it run for 5 minutes earlier just to make sure. In some runs you can for some reason get it to return by killing the process prematurely and you're presented with a massive stream of text.

However, when you start the app with --chat, it works as you'd expect. I was surprised to find that the issue exists in --notebook as well. Start the app with that flag and it just runs indefinitely.

[oobabooga]

Can you see if the error persists with a smaller model like opt-350m?

[StonkGodCapital]

Can you see if the error persists with a smaller model like opt-350m?

opt-350m seems to return an, albeit lengthy, response:

Notebook:

Common sense questions and answers

Question:  What color is the sky?
Factual answer:  The sky has blue, green, red, yellow, orange, white, black, and violet colors.  It's not a color that is used in the sky to indicate a certain location on the earth or for other purposes such as visibility of an object from above.  The blue color is used by astronomers because it is a color that is associated with the blue moon.  It also indicates the moon being near Earth at the time of its appearance.  It can be found in the sky during the night hours when the moon will be visible in the sky.  It is also known as the "moon" color because it occurs only once every two weeks (the first lunar day).  The red color is used to signify the sun's light passing through the clouds.  The yellow color is used by astronomers to denote the moon's light coming through the clouds; it is usually associated with the sun rising over the horizon and then falling back down again.  The purple color is used to

API:

Common sense questions and answers
Question: What color is the sky?
Factual answer: The sky is blue. The blue is a reflection of the sun's light. The blue is the light reflected from the sun's surface, which reflects off the surface of the Earth. The blue is a reflection of the sun's surface, which reflects off the surface of the Earth. The blue is a reflection of the sun's surface, which reflects off the surface of the Earth.
The blue is a reflection of the sun's surface, which reflects off the surface of the Earth. The blue is a reflection of the sun's surface, which reflects off the surface of the Earth. The blue is a reflection of the sun's surface, which reflects off the surface of the Earth. The blue is a reflection of the sun's surface, which reflects off the surface of the Earth. The blue is a reflection of the sun's surface, which reflects off the surface of the Earth. The blue is a reflection of the sun's surface, which reflects off the surface of the Earth.
The

[oobabooga]

It's possible that you are running out of memory with GPT-J-6B.

[StonkGodCapital]

It's possible that you are running out of memory with GPT-J-6B.

The model works fine in chat mode. For some reason the responses aren't capped in the other modes and it just keeps running. The length of text returned is defined by how long I let it run in the times I've been able to get killing the process to spit something out.

[StonkGodCapital]

The issue seems to lie in the eos_token_id figure.

Comparing a chat request to a notebook request, the figure is significantly higher in the notebook request (50K+ vs 198 in chat).

Hardcoding 198 as the figure produces the expected results from both the API and --notebook:

Question: What color is the sky?
Factual answer: The sky is blue.

[github-actions[bot]]

This issue has been closed due to inactivity for 30 days. If you believe it is still relevant, please leave a comment below.