PaddleNLP icon indicating copy to clipboard operation
PaddleNLP copied to clipboard

[Encode Fix]fix ernie-m encode

Open Southpika opened this issue 1 year ago • 1 comments

修正ernie-m系列编码解码结果不一致的问题(异常合并不该合并的token),在原有逻辑中,修改部分为

  1. 保留原有方案中跳过SPIECE_UNDERLINE的操作,实际上应该保留
  2. 如果句首为SPIECE_UNDERLINE,则不应该被加入
  3. 更新相关SPIECE_UNDERLINE的处理逻辑

Southpika avatar Feb 06 '24 08:02 Southpika

Thanks for your contribution!

paddle-bot[bot] avatar Feb 06 '24 08:02 paddle-bot[bot]

This Pull Request is stale because it has been open for 60 days with no activity. 当前Pull Request 60天内无活动,被标记为stale。

github-actions[bot] avatar Apr 07 '24 00:04 github-actions[bot]