PygmalionAI
[Feature]: Implementation of DRY sampler.
🚀 The feature, motivation and pitch
Majority of the bots repeat sentences and due to that, it's annoying.
Implementing DRY sampler for Aphrodite Engine.
DRY is a sampler that forces the model to not repeat while mantaining the quality or even improving it, some of the backends have already support for this.
- Text Gen webui
- koboldai
- llamaccp
Alternatives
No response
Additional context
No response
Might be interesting to add. I currently don't have the bandwidth to tackle this. I may have the time after the 0.5.4 release, but if someone else wants to take over for now, it should be quite easy:
-
Add the parameters related to the DRY sampler here: https://github.com/PygmalionAI/aphrodite-engine/blob/rc_054/aphrodite/common/sampling_params.py
-
Define the sampler's metadata here. e.g. the conditions for triggering it (could be a related sampler param being set to higher than 0, etc) and create the necessary tensors. Plenty of examples here so you won't be lost: https://github.com/PygmalionAI/aphrodite-engine/blob/rc_054/aphrodite/modeling/sampling_metadata.py
-
Finally, implement the actual sampler logic here: https://github.com/PygmalionAI/aphrodite-engine/blob/rc_054/aphrodite/modeling/layers/sampler.py
Should be plenty of examples at each step from other samplers. Please make sure you contribute to the rc_054 branch, unless you're reading this in a few weeks and that branch is merged into main.
I've opened an draft PR but it definitely has some issues around the sequence breakers: https://github.com/PygmalionAI/aphrodite-engine/pull/827
I'm not familiar enough with how the sampling metadata works to correctly add a new value that's a List[int] as opposed to a scalar value like most of the existing parameters, added some notes
