KoboldAI
Generating text after a comma always starts with the words Ice or iced using Janeway
https://user-images.githubusercontent.com/37173361/162937668-d7169cb1-cce3-4bab-9434-fd52bd095a77.mp4
Example video
Which version of KoboldAI and Janeway are you using? Another thing to keep in mind is that for all AI models depending on settings they can generate very similar sentences on retry's. You can increase the temperature if its to samey.
Whatever the latest version in the Google colab is, and I'm using Janeway official.
And yes I'll give it a shot when I'm back at my PC, haven't really messed too much with the settings yet, thanks!
Similar behavior is happening here : https://github.com/huggingface/transformers/issues/17545 So it could potentially be an issue with the model or HF implementation, ill leave it open for now so we can track their progress better.
Got it for the first time myself while trying to diagnose another issue, it is definitely not always and the replication steps are currently unkown. The moment it happens however it keeps happening with extreme probability and the entire probability list does not make sense.
Here is some debug output: 15.191456 38.304% 3711 [iced] 14.198950 14.197% 2575 [urch] 13.868835 10.206% 488 [ich] 13.187734 5.165% 1834 [urs] 12.494523 2.582% 1134 [ik] 12.393734 2.335% 5523 [Â] 11.894249 1.417% 257 [ a] 11.807496 1.299% 4557 [**] 11.792485 1.280% 624 [ick] 11.691641 1.157% 364 [ers]
Another common one with this bug is urchin, so this bug definitely still exists somewhere. The question is what triggers the behavior. This was on the TPU, so no HF implementation.