Ikram Ali comments

Results 9 comments of


                                            Ikram Ali

Issue in Installation

Urduhack support python 3.6 and above.

Normalize ﷲ to الله

Ok. I will look into this.

Normalization and word tokenization have issues.

@mirfan899 will look into this issue.

[FEATURE] Support Python 3.10

Any progress on this?

Raise warning for OOV tokens in fast tokenizer fix #29159

> BTW @akkefa before we review the PR, could you make sure you approve circle CI? @ArthurZucker How I can approve circle CI flow?

[tokenizer] Inconsistent behavior in slow tokenizer and fast tokenizer

@ArthurZucker @Ki-Seki Has it been fixed yet? I want to start working on it.

[tokenizer] Inconsistent behavior in slow tokenizer and fast tokenizer

@ArthurZucker What should the behavior be when both tokenizer types encounter an OOV token? Should it simply raise an index error exception, or do you have something else in mind?

[tokenizer] Inconsistent behavior in slow tokenizer and fast tokenizer

@ArthurZucker We can start a warning about this new change. After a certain version release, we can introduce a custom error exception for OOV word in tokenizer.

[tokenizer] Inconsistent behavior in slow tokenizer and fast tokenizer

@ArthurZucker My plan is to start raising a warning in the fast tokenizer implementation so that users know when out-of-vocabulary (OOV) tokens are being ignored. In the normal tokenizer, OOV...