sudachi.rs icon indicating copy to clipboard operation
sudachi.rs copied to clipboard

Published 20 hours ago • verified publisher icon WorksApplications

create pre-tokenizer with surface-projection does not override dictionary-projection

Open mh-northlander opened this issue 4 weeks ago • 0 comments

Creating pre-tokenizer with surface-projection specified does not overrides the projection of dictionary.

In test: https://github.com/WorksApplications/sudachi.rs/blob/232d9ee09e8397f5dfaa6f10c743cf09cf1d4f57/python/tests/test_pretokenizers.py#L105-L119

Is this intentional or bug?

mh-northlander avatar Jun 10 '24 00:06 mh-northlander