PaddleNLP
PaddleNLP copied to clipboard
[Tokenizer]Convert fast_tokenizer to hf-tokenizers
迁移应用hf-tokenizers作为fast-tokenizer
除了tokenizer相关添加以外修复了一些小问题
-
name_or_path
属性没有正确赋值 - 修复
from_slow
参数没有被使用的问题
注:目前ERNIE-M slow与fast版本的结果存在diff,待确定使用版本
Thanks for your contribution!
Codecov Report
Attention: Patch coverage is 82.29167%
with 119 lines
in your changes missing coverage. Please review.
Project coverage is 55.40%. Comparing base (
bc91dc6
) to head (eeb1c5c
). Report is 170 commits behind head on develop.
:exclamation: Current head eeb1c5c differs from pull request most recent head 66575bb
Please upload reports for the commit 66575bb to get more accurate results.
Additional details and impacted files
@@ Coverage Diff @@
## develop #7974 +/- ##
===========================================
+ Coverage 55.25% 55.40% +0.14%
===========================================
Files 613 621 +8
Lines 95625 96168 +543
===========================================
+ Hits 52837 53280 +443
- Misses 42788 42888 +100
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
This Pull Request is stale because it has been open for 60 days with no activity. 当前Pull Request 60天内无活动,被标记为stale。