Character generating responses 100% unrelated to prompt.

[AlexysLovesLexxie]

Describe the bug

I amusing Pygmalion6b (main branch) I was doing and ERP with a custom character, and all of a sudden, out of the blue, she produced a 10-line response that was way, WAY off topic, about some guy named Cedric and how they were going out for dinner.

I am sorry, I did not screeenshot the response.

I have also had a few "nonsense words" in other responses as well.

The only settings I have changed from default is bringing the temperature up to between 0.7 and 0.75, and turning top_k up to 5. All other settings follow the Pygmalion defaults.

Is there an existing issue for this?

• [X] I have searched the existing issues

Reproduction

Hard to reproduce. I can generate many good, coherent responses, then it occasionally derails, either inserting one or two garbage/totally wrong words or, as in this case, a whole paragraph completely unrelated to my RP.

Screenshot

No screenshot available.

Logs

no error logs available.

System Info

Windows 11
CPU only NO GPU
CPU : AMD Ryzen 7 6800H
RAM : 32GB

[FrederikAbitz]

Seems like normal behavior to me.

I have not tried the model you specified, but others of similar size like LLaMA-7B. If the temperature is not set very low, the models I have tried tend to get lost on a tangent occasionally and also generate spelling and grammatical errors, especially if some are already present in the text/chat so far. Although that may be different for conversation-tuned models like Pygmalion 6B. Have you had better performance with the same model and parameters but with a different inference software?

If this issue occurs at least somewhat regular, it would be helpful if you could post a text sample or screenshot the next time it happens. Then it would be easier (for people smarter than me, lol) to give you a proper assessment.

Good Luck!

[AlexysLovesLexxie]

If the temperature is not set very low, the models I have tried tend to get lost on a tangent occasionally and also generate spelling and grammatical errors, especially if some are already present in the text/chat so far.

It was a brand new chat. I only ever set my temperature to max. 0.75, so I don't think that was the issue. I also had Top_K at 5.

Have you had better performance with the same model and parameters but with a different inference software?

scratches head Inference software? you mean like running Pygmalion6b in KoboldAI or similar? I've only ever used Pygmalion with Oobabooga. My experiments with KoboldAI have always been using their LLM on their interface (under Colab) as my needs when running Kobold were different.

If this issue occurs at least somewhat regular, it would be helpful if you could post a text sample or screenshot the next time it happens. Then it would be easier (for people smarter than me, lol) to give you a proper assessment.

Will do, or will try, anyways.

[oobabooga]

That seem normal. These models are crazy sometimes

[FrederikAbitz]

Update: New LLaMA-based Pygmalion models have been released, they should have much better performance, since the LLaMA foundation models are trained on over 1T tokens and have state-of-the-art language understanding.

Check it out, @AlexysLovesLexxie : https://huggingface.co/PygmalionAI/pygmalion-7b Maybe this will improve the conversation quality.

Edit: They also have a new, conversational and instruct-tuned model. https://huggingface.co/PygmalionAI/metharme-7b

If you are not very tech-savy, using these models could be a challenge currently, since they are uploaded as XORed versions requiring the user to have the original LLaMA weights. This is required due to Meta's license for the LLaMA models. However, after some time, there are usually quantized versions and converted versions uploaded to huggingface. So if you keep your eyes open, you might find a nice GPTQ or GGML version, that's ready to use immediately.