rten
rten copied to clipboard
robertknight
Support Llama 3 tokenizer (implement `ignore_merges` behavior)
Attempting to load the tokenizer.json file from Llama 3.2 fails with an error processing the BPE merge entries:
Error: BpeError(InvalidMergeEntry("Ġ ĠĠĠ"))
If rten-text is modified to ignore this error, then the qwen2_chat example works with Llama 3.2, after a minor modification to the special token IDs.
Edit: I have just noticed the ignore_merges: true in the tokenizer.json file. This seems relevant.
ignore_merges was added in https://github.com/huggingface/tokenizers/commit/914576f7edcba7116b3d22fd0f2b8f727d652c3a. See also https://github.com/huggingface/tokenizers/pull/1493/files.
The documentation says:
ignore_merges (bool, optional) — Whether or not to match tokens with the vocab before using merges.
It turns out that after fixing the data structures used by the BPE encoder in https://github.com/robertknight/rten/pull/1041, ignore_merges support is not critical to getting a basic Llama 3 example working, though it will still be needed.
The llama3 tokenizer's vocabulary has 588 tokens out of 128000, or <0.5%, where the bytes are valid UTF-8 and the token cannot be obtained by iteratively merging tokens according to the BPE merge process and the tokenizer's merge list.
Additionally there are 1352 tokens where the bytes are not valid UTF-8 that I couldn't test because Bpe::encode_piece requires valid UTF-8 input. These will never be produced by the string splitting process because the pre-tokenizer's output are always valid strings.
token=rodní encoded_token=rodnÃŃ expected=106920 actual=[299, 17851, 2483]
token= заболевания encoded_token=ĠзаболеваниÑı expected=110563 actual=[101447, 7975, 33742, 39280]
token=ıldı encoded_token=ıldı expected=102564 actual=[3862, 509, 3862]
token= ortalama encoded_token=Ġortalama expected=124917 actual=[220, 34472, 3105]
token= gerektir encoded_token=Ġgerektir expected=120260 actual=[67387, 5964, 404]
token=maması encoded_token=maması expected=123107 actual=[76, 29189, 3862]
token= Cumhuriyet encoded_token=ĠCumhuriyet expected=118972 actual=[32678, 71, 6198, 47492]
token= almaktadır encoded_token=Ġalmaktadır expected=122318 actual=[90584, 10114, 101252]
token= mnoha encoded_token=Ġmnoha expected=121091 actual=[296, 2201, 4317]
token= Müslüman encoded_token=ĠMüslüman expected=119959 actual=[64730, 3306, 2448, 1543]
token= Savaşı encoded_token=ĠSavaÅŁÄ± expected=121878 actual=[328, 2979, 78413]
token= chiều encoded_token=Ġchiá»ģu expected=111317 actual=[26883, 41038, 84]
token= ünlü encoded_token=Ġünlü expected=124509 actual=[10709, 16017, 2448]
token= siyaset encoded_token=Ġsiyaset expected=124419 actual=[4502, 88, 5136]
token= раніше encoded_token=ĠÑĢанÑĸÑĪе expected=122853 actual=[18600, 101262, 60009]
token= iletişim encoded_token=ĠiletiÅŁim expected=117380 actual=[31905, 10462, 7370, 318]
token= наличие encoded_token=ĠналиÑĩие expected=125791 actual=[13373, 91216, 50306]
token= oluştur encoded_token=ĠoluÅŁtur expected=104700 actual=[8492, 84, 48497, 324]
token= внеш encoded_token=ĠвнеÑĪ expected=112759 actual=[5927, 2156, 47067]
token= açıklam encoded_token=Ġaçıklam expected=113720 actual=[64010, 91166, 309]
token=ujícím encoded_token=ujÃŃcÃŃm expected=116552 actual=[9832, 30515, 56094]
token= tekrar encoded_token=Ġtekrar expected=108531 actual=[1028, 10056, 277]
token= визнач encoded_token=ĠвизнаÑĩ expected=107996 actual=[5927, 41873, 31281]
token= yeşil encoded_token=ĠyeÅŁil expected=126773 actual=[20043, 7370, 321]
token=ujících encoded_token=ujÃŃcÃŃch expected=115614 actual=[9832, 30515, 100489]
token= розвит encoded_token=ĠÑĢозвиÑĤ expected=105672 actual=[18600, 49855, 9542]
token= анализ encoded_token=Ġанализ expected=122409 actual=[21022, 14525, 102921]
token= mnoh encoded_token=Ġmnoh expected=108294 actual=[296, 2201, 71]
token= našeho encoded_token=ĠnaÅ¡eho expected=127101 actual=[112819, 2701, 78]
token= позволя encoded_token=ĠпозволÑı expected=109689 actual=[5173, 49855, 96485]
token= zahrn encoded_token=Ġzahrn expected=123745 actual=[1167, 15464, 77]
token=ilmektedir encoded_token=ilmektedir expected=108577 actual=[107567, 6702, 404]
token= nákup encoded_token=Ġnákup expected=123639 actual=[308, 72542, 455]
token= позвол encoded_token=Ġпозвол expected=121267 actual=[5173, 49855, 7975]
token= اطلاعات encoded_token=ĠاطÙĦاعات expected=105370 actual=[102992, 82070, 105462]
token= uygulama encoded_token=Ġuygulama expected=120009 actual=[102983, 360, 3105]
token= bağımsız encoded_token=Ġbaģımsız expected=119333 actual=[13081, 44386, 1026, 48980]
token= канди encoded_token=Ġканди expected=120100 actual=[7820, 76559, 1840]
token= sahibi encoded_token=Ġsahibi expected=112304 actual=[829, 5923, 72]
token= vítěz encoded_token=ĠvÃŃtÄĽz expected=119403 actual=[348, 16485, 22161, 89]
token= döneminde encoded_token=Ġdöneminde expected=119419 actual=[101407, 26768, 451]
token=anglicky encoded_token=anglicky expected=126715 actual=[526, 1228, 88]
token= використ encoded_token=ĠвикоÑĢиÑģÑĤ expected=103754 actual=[107038, 9239, 35111]
token=iyordu encoded_token=iyordu expected=104121 actual=[16618, 541, 84]
token= değişiklik encoded_token=ĠdeÄŁiÅŁiklik expected=115976 actual=[409, 44907, 7370, 1609, 7792]
token= професси encoded_token=ĠпÑĢоÑĦеÑģÑģи expected=116983 actual=[103665, 94520, 1840]
token= Hiệp encoded_token=ĠHiá»ĩp expected=124254 actual=[21694, 26298, 79]
token=ประมาณ encoded_token=à¸Ľà¸£à¸°à¸¡à¸²à¸ĵ expected=115164 actual=[100694, 103828, 87448]
token= odbor encoded_token=Ġodbor expected=110107 actual=[297, 2042, 269]
token= принадлеж encoded_token=ĠпÑĢинадлеж expected=126778 actual=[107279, 23680, 112833]
token= граду encoded_token=ĠгÑĢадÑĥ expected=120196 actual=[25756, 118900, 3865]
token= fırsat encoded_token=Ġfırsat expected=120538 actual=[282, 3862, 5544, 266]
token= množství encoded_token=ĠmnožstvÃŃ expected=115016 actual=[296, 2201, 12453, 101707]
token= zdraví encoded_token=ĠzdravÃŃ expected=119785 actual=[1167, 3696, 402, 2483]
token=jících encoded_token=jÃŃcÃŃch expected=120959 actual=[73, 30515, 100489]
token= развития encoded_token=ĠÑĢазвиÑĤиÑı expected=111643 actual=[39479, 104169, 40231]
token= kurtar encoded_token=Ġkurtar expected=121357 actual=[597, 5757, 277]
token= zvlášt encoded_token=Ġzvlášt expected=117909 actual=[1167, 14878, 1995, 85578]
token= girişim encoded_token=ĠgiriÅŁim expected=120663 actual=[342, 21336, 7370, 318]
token= vystav encoded_token=Ġvystav expected=127671 actual=[348, 599, 402]
token= hỏi encoded_token=Ġhá»ıi expected=107193 actual=[305, 86242, 72]
token= yaptığ encoded_token=Ġyaptıģ expected=119776 actual=[379, 2756, 93687]
token= Českosloven encoded_token=ĠÄĮeskosloven expected=123611 actual=[122587, 57778, 107394]
token= karşılık encoded_token=ĠkarÅŁÄ±lık expected=124621 actual=[101336, 53271, 74]
token= žádný encoded_token=Ġžádný expected=123178 actual=[32063, 1995, 17851, 20195]
token=ıyordu encoded_token=ıyordu expected=107102 actual=[100500, 541, 84]
token= tavsiye encoded_token=Ġtavsiye expected=126740 actual=[93309, 6455, 9188]
token= chtěl encoded_token=ĠchtÄĽl expected=122428 actual=[272, 427, 100599]
token= препараты encoded_token=ĠпÑĢепаÑĢаÑĤÑĭ expected=125873 actual=[104567, 17721, 110102]
token= Việt encoded_token=ĠViá»ĩt expected=101798 actual=[11655, 26298, 83]
token= prvním encoded_token=ĠprvnÃŃm expected=122268 actual=[550, 29238, 56094]
token= začátku encoded_token=ĠzaÄįátku expected=124352 actual=[103393, 1995, 6375, 84]
token= việc encoded_token=Ġviá»ĩc expected=100769 actual=[3355, 26298, 66]
token= суще encoded_token=ĠÑģÑĥÑīе expected=126984 actual=[5524, 43293, 1532]
token= tespit encoded_token=Ġtespit expected=114120 actual=[259, 25632, 275]
token=илася encoded_token=илаÑģÑı expected=124292 actual=[64292, 18437, 4329]
token=しない encoded_token=ãģĹãģªãģĦ expected=113878 actual=[21105, 103, 16995]
token= ayrıntı encoded_token=Ġayrıntı expected=117325 actual=[110221, 406, 3862]
token= geliyor encoded_token=Ġgeliyor expected=115979 actual=[342, 12574, 48320]
token= аналог encoded_token=Ġаналог expected=119477 actual=[21022, 14525, 3114, 14837]
token= zahrani encoded_token=Ġzahrani expected=115326 actual=[1167, 15464, 5676]
token= tüket encoded_token=Ġtüket expected=112668 actual=[259, 2448, 26554]
token= tiểu encoded_token=Ġtiá»ĥu expected=113134 actual=[9165, 34550, 84]
token= Александр encoded_token=ĠÐIJлекÑģандÑĢ expected=124610 actual=[124877, 7486, 63873]
token= підтрим encoded_token=ĠпÑĸдÑĤÑĢим expected=112620 actual=[100621, 101231, 16494]
token= zprávy encoded_token=Ġzprávy expected=124465 actual=[1167, 652, 1995, 14029]
token=lıklı encoded_token=lıklı expected=115129 actual=[26693, 10784, 3862]
token= jednodu encoded_token=Ġjednodu expected=114178 actual=[18806, 35666, 84]
token= přítom encoded_token=ĠpÅĻÃŃtom expected=120894 actual=[59406, 16485, 316]
token= kuchy encoded_token=Ġkuchy expected=123510 actual=[597, 1412, 88]
token= vzdálen encoded_token=Ġvzdálen expected=124843 actual=[348, 49755, 1995, 2963]
token= Merkez encoded_token=ĠMerkez expected=120102 actual=[8930, 441, 89]
token=ektedir encoded_token=ektedir expected=101673 actual=[1247, 6702, 404]
token=ılmış encoded_token=ılmÄ±ÅŁ expected=104516 actual=[3862, 18244, 33573]
token= peníze encoded_token=ĠpenÃŃze expected=119243 actual=[5869, 2483, 3059]
token= kırmızı encoded_token=Ġkırmızı expected=125171 actual=[109338, 8892, 104658]
token= nhiêu encoded_token=Ġnhiêu expected=121135 actual=[308, 6151, 102069]
token= smlou encoded_token=Ġsmlou expected=108980 actual=[274, 1029, 283]
token= ابراه encoded_token=ĠابراÙĩ expected=124971 actual=[13258, 84927, 100308]
token= pravidel encoded_token=Ġpravidel expected=116940 actual=[550, 15567, 301]
token=алась encoded_token=алаÑģÑĮ expected=105435 actual=[16331, 18437, 4929]
token=алася encoded_token=алаÑģÑı expected=108918 actual=[16331, 18437, 4329]
token= siyasi encoded_token=Ġsiyasi expected=117798 actual=[4502, 88, 10426]
token= відріз encoded_token=ĠвÑĸдÑĢÑĸз expected=125759 actual=[106079, 63873, 100722]
token= zdravot encoded_token=Ġzdravot expected=109414 actual=[1167, 3696, 402, 354]
token= Özellikle encoded_token=ĠÃĸzellikle expected=120946 actual=[35137, 98329, 1609, 273]
token= údajů encoded_token=Ġúdajů expected=115002 actual=[18448, 67, 1662, 52353]
token= октября encoded_token=ĠокÑĤÑıбÑĢÑı expected=119288 actual=[72664, 101331, 104663]
token= огранич encoded_token=ĠогÑĢаниÑĩ expected=115977 actual=[105380, 2233, 14332, 6148]
token= воздуха encoded_token=ĠвоздÑĥÑħа expected=117955 actual=[5927, 35387, 101082, 1506]
token= Việc encoded_token=ĠViá»ĩc expected=114195 actual=[11655, 26298, 66]
token= İslam encoded_token=ĠÄ°slam expected=114848 actual=[39521, 3306, 309]
token= воздейств encoded_token=ĠвоздейÑģÑĤв expected=114540 actual=[5927, 35387, 77628]
token= визнача encoded_token=ĠвизнаÑĩа expected=119162 actual=[5927, 41873, 31281, 1506]
token= olmuştur encoded_token=ĠolmuÅŁtur expected=115040 actual=[8492, 15479, 48497, 324]
token= điều encoded_token=ĠÄijiá»ģu expected=101309 actual=[74686, 41038, 84]
token= вполне encoded_token=Ġвполне expected=122812 actual=[101996, 47680, 1532]
token= профессиональ encoded_token=ĠпÑĢоÑĦеÑģÑģионалÑĮ expected=121209 actual=[103665, 94520, 1840, 101811]
token=müştür encoded_token=mÃ¼ÅŁtür expected=122315 actual=[76, 2448, 48497, 5297]
token= yaptığı encoded_token=Ġyaptıģı expected=110486 actual=[379, 2756, 75437]
token= zástup encoded_token=Ġzástup expected=117749 actual=[1167, 56503, 455]
token= електрон encoded_token=ĠелекÑĤÑĢон expected=122951 actual=[22918, 3114, 30656, 103710]
token= yabancı encoded_token=Ġyabancı expected=116014 actual=[379, 370, 1149, 3862]
token= Ngoài encoded_token=ĠNgoÃłi expected=114031 actual=[452, 3427, 53651]
token= Insecta encoded_token=ĠInsecta expected=114843 actual=[763, 9885, 64]
token= činnosti encoded_token=ĠÄįinnosti expected=117347 actual=[33902, 6258, 71779]
token= розвитку encoded_token=ĠÑĢозвиÑĤкÑĥ expected=107695 actual=[18600, 49855, 9542, 36352]
token= mutlaka encoded_token=Ġmutlaka expected=121973 actual=[5318, 75, 13637]
token= yüzyıl encoded_token=Ġyüzyıl expected=113201 actual=[379, 2448, 4341, 55888]
token= предназнач encoded_token=ĠпÑĢедназнаÑĩ expected=126357 actual=[57925, 2156, 19188, 31281]
token=назнач encoded_token=назнаÑĩ expected=124593 actual=[2156, 19188, 31281]
token= нанес encoded_token=ĠнанеÑģ expected=126457 actual=[6850, 7486, 23297]
token=üyordu encoded_token=üyordu expected=115970 actual=[100565, 541, 84]
token=ımsız encoded_token=ımsız expected=114185 actual=[3862, 1026, 48980]
token= людини encoded_token=ĠлÑİдини expected=111367 actual=[94136, 71438, 1840]
token=dıkları encoded_token=dıkları expected=126754 actual=[67, 91166, 68228]
token= merkezi encoded_token=Ġmerkezi expected=116627 actual=[4809, 441, 8510]
token= hiểu encoded_token=Ġhiá»ĥu expected=105744 actual=[15960, 34550, 84]
token= вияви encoded_token=ĠвиÑıви expected=127660 actual=[5927, 40231, 100324]
token= позволяет encoded_token=ĠпозволÑıеÑĤ expected=115091 actual=[5173, 49855, 96485, 8341]
token= характ encoded_token=ĠÑħаÑĢакÑĤ expected=103434 actual=[45658, 1506, 123282, 1830]
token=ımlar encoded_token=ımlar expected=127748 actual=[3862, 1029, 277]
token= знаком encoded_token=Ġзнаком expected=123279 actual=[105161, 16248, 12507]
token= фундамент encoded_token=ĠÑĦÑĥндаменÑĤ expected=127417 actual=[113896, 127518, 19190]
token= اروپ encoded_token=ĠارÙĪÙ¾ expected=116144 actual=[13258, 86096, 68483]
token= بیرون encoded_token=ĠبÛĮرÙĪÙĨ expected=110785 actual=[102363, 86096, 12061]
token= качестве encoded_token=ĠкаÑĩеÑģÑĤве expected=110734 actual=[7820, 50223, 35034, 1532]
token=ırlar encoded_token=ırlar expected=124703 actual=[3862, 2438, 277]
token= přiroz encoded_token=ĠpÅĻiroz expected=126634 actual=[59406, 8869, 89]
token= İslâm encoded_token=ĠÄ°slâm expected=124078 actual=[39521, 3306, 87828]
token= случаях encoded_token=ĠÑģлÑĥÑĩаÑıÑħ expected=115183 actual=[90226, 36497, 10693]
token= klíč encoded_token=ĠklÃŃÄį expected=126295 actual=[20839, 2483, 13453]
token= případech encoded_token=ĠpÅĻÃŃpadech expected=125637 actual=[100578, 79, 1037, 331]
token= Mühendis encoded_token=ĠMühendis expected=124961 actual=[386, 22284, 408, 285]
token= nhiên encoded_token=Ġnhiên expected=102536 actual=[308, 6151, 28119]
token=kyně encoded_token=kynÄĽ expected=119709 actual=[74, 1910, 22161]
token= daleko encoded_token=Ġdaleko expected=127232 actual=[294, 1604, 9509]
token= architekt encoded_token=Ġarchitekt expected=126316 actual=[5438, 635, 5964]
token=ávající encoded_token=ávajÃŃcÃŃ expected=118508 actual=[100350, 1662, 122703]
token= durumlarda encoded_token=Ġdurumlarda expected=127202 actual=[10878, 43331, 72626]
token= milyon encoded_token=Ġmilyon expected=107667 actual=[296, 1570, 263]
token= neredeyse encoded_token=Ġneredeyse expected=122551 actual=[308, 12616, 117077]
token=itelné encoded_token=itelné expected=124647 actual=[275, 17912, 978]
token=дивиду encoded_token=дивидÑĥ expected=118805 actual=[111506, 48262, 3865]
token= ткани encoded_token=ĠÑĤкани expected=119917 actual=[11047, 4898, 14332]
token= yıldır encoded_token=Ġyıldır expected=127765 actual=[122231, 509, 30333]
token=vající encoded_token=vajÃŃcÃŃ expected=126173 actual=[85, 1662, 122703]
token=Việc encoded_token=Viá»ĩc expected=125059 actual=[36644, 26298, 66]
token= dispozici encoded_token=Ġdispozici expected=119322 actual=[834, 123647, 3457]
token= každý encoded_token=Ġkaždý expected=112357 actual=[16909, 12453, 67, 20195]
token= bulunduğu encoded_token=ĠbulunduÄŁu expected=114991 actual=[7173, 1263, 84, 102195]
token= látky encoded_token=Ġlátky expected=127968 actual=[54738, 6375, 88]
token= Králové encoded_token=ĠKrálové expected=127034 actual=[16852, 1995, 36247, 978]
token= علاوه encoded_token=ĠعÙĦاÙĪÙĩ expected=121680 actual=[45082, 82070, 101683]
token= ücretsiz encoded_token=Ġücretsiz expected=122883 actual=[10709, 53810, 450]
token= декабря encoded_token=ĠдекабÑĢÑı expected=118793 actual=[107614, 38722, 100674]
token= yöntem encoded_token=Ġyöntem expected=109693 actual=[98513, 406, 336]
token= založ encoded_token=Ġzalož expected=111962 actual=[1167, 12812, 12453]
token= розвиток encoded_token=ĠÑĢозвиÑĤок expected=123422 actual=[18600, 49855, 9542, 15088]
token= toplumsal encoded_token=Ġtoplumsal expected=125143 actual=[105706, 6370, 278]
token= görmek encoded_token=Ġgörmek expected=124459 actual=[31285, 8892, 1247]
token= электри encoded_token=ĠÑįлекÑĤÑĢи expected=119201 actual=[21599, 3114, 30656, 31203]
token=abancı encoded_token=abancı expected=111691 actual=[370, 1149, 3862]
token= аналіз encoded_token=ĠаналÑĸз expected=116546 actual=[21022, 14525, 100667, 9136]
token= располож encoded_token=ĠÑĢаÑģполож expected=111750 actual=[78876, 102500, 21956]
token= المدينة encoded_token=ĠاÙĦÙħدÙĬÙĨØ© expected=121936 actual=[54579, 80920, 101788]
token= เพราะ encoded_token=Ġà¹Ģà¸ŀราะ expected=118425 actual=[95582, 103058, 102318]
token= наличии encoded_token=ĠналиÑĩии expected=121080 actual=[13373, 91216, 47273]
token= kaldır encoded_token=Ġkaldır expected=109009 actual=[597, 4852, 30333]
token= развитие encoded_token=ĠÑĢазвиÑĤие expected=123398 actual=[39479, 104169, 50306]
token= характеристи encoded_token=ĠÑħаÑĢакÑĤеÑĢиÑģÑĤи expected=111949 actual=[45658, 1506, 123282, 58060, 35111, 1840]
token= використовувати encoded_token=ĠвикоÑĢиÑģÑĤовÑĥваÑĤи expected=124336 actual=[107038, 9239, 35111, 6856, 101539]
token= nhanh encoded_token=Ġnhanh expected=105314 actual=[308, 10118, 71]
token= ослож encoded_token=ĠоÑģлож expected=127865 actual=[8379, 41036, 21956]
token= obvykle encoded_token=Ġobvykle expected=122301 actual=[1536, 14029, 23960]
token=sahuje encoded_token=sahuje expected=124393 actual=[82, 27793, 3841]
token= Örneğin encoded_token=ĠÃĸrneÄŁin expected=121744 actual=[35137, 67020, 104453]
token= staveb encoded_token=Ġstaveb expected=121563 actual=[357, 525, 65]
token= mücadele encoded_token=Ġmücadele expected=120338 actual=[59385, 33896, 273]
token= Müdürlüğü encoded_token=ĠMüdürlÃ¼ÄŁÃ¼ expected=125302 actual=[64730, 102234, 2438, 2448, 104998]
token= hudeb encoded_token=Ġhudeb expected=121818 actual=[305, 799, 65]
token= zdrav encoded_token=Ġzdrav expected=105299 actual=[1167, 3696, 402]
token= مدير encoded_token=ĠÙħدÙĬر expected=123913 actual=[24252, 80920, 11318]
token= зависим encoded_token=ĠзавиÑģим expected=113077 actual=[93323, 13810, 16494]
token= sektör encoded_token=Ġsektör expected=108921 actual=[513, 5964, 9603]
token=ーション encoded_token=ãĥ¼ãĤ·ãĥ§ãĥ³ expected=114836 actual=[38248, 115, 78205, 16073]
token= کلاس encoded_token=ĠÚ©ÙĦاس expected=122059 actual=[46693, 82070, 20665]
token=rální encoded_token=rálnÃŃ expected=125700 actual=[38468, 2312, 2483]
token= kültür encoded_token=Ġkültür expected=110383 actual=[67048, 4937, 5297]
token= zvyš encoded_token=ĠzvyÅ¡ expected=123563 actual=[1167, 14029, 11906]
token= yönelik encoded_token=Ġyönelik expected=110692 actual=[98513, 8301, 1609]
token= söyledi encoded_token=Ġsöyledi expected=110143 actual=[51387, 7756, 72]
token= nhiều encoded_token=Ġnhiá»ģu expected=100937 actual=[308, 6151, 41038, 84]
token= dikkat encoded_token=Ġdikkat expected=106350 actual=[294, 30987, 266]
token= birey encoded_token=Ġbirey expected=114283 actual=[293, 556, 88]
token=jícím encoded_token=jÃŃcÃŃm expected=127954 actual=[73, 30515, 56094]
token=üncü encoded_token=üncü expected=110728 actual=[2448, 1031, 2448]
token= nejsou encoded_token=Ġnejsou expected=113860 actual=[841, 2580, 283]
token= приступ encoded_token=ĠпÑĢиÑģÑĤÑĥп expected=120978 actual=[12561, 35111, 49907]
token= розви encoded_token=ĠÑĢозви expected=127719 actual=[18600, 49855, 1840]
token= kuruluş encoded_token=ĠkuruluÅŁ expected=111253 actual=[28364, 25415, 7370]
token=україн encoded_token=ÑĥкÑĢаÑĹн expected=125265 actual=[67187, 22911, 100268, 2156]
token=Türkiye encoded_token=Türkiye expected=112353 actual=[51, 5297, 96303]
token= bulundu encoded_token=Ġbulundu expected=123416 actual=[7173, 1263, 84]
token= vzdělávání encoded_token=ĠvzdÄĽlávánÃŃ expected=121662 actual=[348, 49755, 100599, 106832]
token= Fransız encoded_token=ĠFransız expected=120405 actual=[2939, 598, 48980]
token= vardı encoded_token=Ġvardı expected=108497 actual=[348, 569, 3862]
token= yıllık encoded_token=Ġyıllık expected=115722 actual=[122231, 657, 41990]
token=méně encoded_token=ménÄĽ expected=115269 actual=[76, 10610, 22161]
token= yüksel encoded_token=Ġyüksel expected=108866 actual=[379, 2448, 2857, 301]
token=ватися encoded_token=ваÑĤиÑģÑı expected=127994 actual=[105495, 13810, 4329]
token= ngoại encoded_token=Ġngoại expected=109621 actual=[308, 3427, 38159]
token= doğrudan encoded_token=ĠdoÄŁrudan expected=120525 actual=[94382, 67455, 276]
token= küçük encoded_token=Ġküçük expected=106527 actual=[67048, 3209, 40957]
token=uştur encoded_token=uÅŁtur expected=113234 actual=[84, 48497, 324]
token= kiếm encoded_token=Ġkiếm expected=111365 actual=[20548, 27160, 76]
token=аракт encoded_token=аÑĢакÑĤ expected=103003 actual=[1506, 123282, 1830]
token= písem encoded_token=ĠpÃŃsem expected=127213 actual=[281, 2483, 7143]
token= özellikleri encoded_token=Ġözellikleri expected=125748 actual=[17372, 98329, 115763]
token=uyordu encoded_token=uyordu expected=112907 actual=[4168, 541, 84]
token= množ encoded_token=Ġmnož expected=112354 actual=[296, 2201, 12453]
token= даже encoded_token=Ġдаже expected=104199 actual=[7952, 38657, 1532]
token=Ngoài encoded_token=NgoÃłi expected=118795 actual=[45, 3427, 53651]
token=iyesi encoded_token=iyesi expected=114767 actual=[72, 9891, 72]
token= králov encoded_token=Ġkrálov expected=115487 actual=[23975, 1995, 36247]
token= değişik encoded_token=ĠdeÄŁiÅŁik expected=108564 actual=[409, 44907, 7370, 1609]
token= gerekiyor encoded_token=Ġgerekiyor expected=115431 actual=[67387, 6780, 48320]
token= tarihi encoded_token=Ġtarihi expected=109133 actual=[259, 2850, 6151]
token= заяви encoded_token=ĠзаÑıви expected=119054 actual=[11122, 36497, 100324]
token= zejména encoded_token=Ġzejména expected=111493 actual=[14017, 67039, 978, 3458]
token= eylem encoded_token=Ġeylem expected=118699 actual=[384, 982, 76]
token= problémy encoded_token=Ġproblémy expected=122410 actual=[52088, 978, 2465]
token= republika encoded_token=Ġrepublika expected=124691 actual=[2109, 392, 747, 4657]
token= профилакти encoded_token=ĠпÑĢоÑĦилакÑĤи expected=127024 actual=[103665, 64292, 16248, 23311]
token= dolaş encoded_token=ĠdolaÅŁ expected=121764 actual=[294, 8083, 7370]
token= воздух encoded_token=ĠвоздÑĥÑħ expected=119725 actual=[5927, 35387, 101082]
token= biểu encoded_token=Ġbiá»ĥu expected=106329 actual=[6160, 34550, 84]
token=ılmıştır encoded_token=ılmÄ±ÅŁtır expected=114860 actual=[3862, 18244, 106232]
token= Dünya encoded_token=ĠDünya expected=110920 actual=[423, 2448, 23741]
token= Odkazy encoded_token=ĠOdkazy expected=119655 actual=[507, 7737, 13933]
token= дозволя encoded_token=ĠдозволÑı expected=118751 actual=[7952, 49855, 96485]
token= уменьш encoded_token=ĠÑĥменÑĮÑĪ expected=121235 actual=[14257, 38972, 4929, 12426]
token= sordu encoded_token=Ġsordu expected=124151 actual=[274, 541, 84]
token= nhiễm encoded_token=Ġnhiá»ħm expected=114429 actual=[308, 6151, 103490, 76]
token=пеки encoded_token=пеки expected=119000 actual=[8164, 15298, 1840]
token= dikkate encoded_token=Ġdikkate expected=125697 actual=[294, 30987, 349]
token= tisíc encoded_token=ĠtisÃŃc expected=117352 actual=[259, 285, 30515]
token=ılmaktadır encoded_token=ılmaktadır expected=126545 actual=[3862, 18244, 10114, 101252]
token= předsed encoded_token=ĠpÅĻedsed expected=118228 actual=[59406, 6910, 291]
token= rahatsız encoded_token=Ġrahatsız expected=121035 actual=[436, 1494, 1900, 48980]
token= olsun encoded_token=Ġolsun expected=110642 actual=[297, 4835, 359]
token= Fakültesi encoded_token=ĠFakültesi expected=120458 actual=[112413, 36624, 2392, 72]
token= İtalya encoded_token=ĠÄ°talya expected=123468 actual=[124565, 5893, 64]
token= января encoded_token=ĠÑıнваÑĢÑı expected=117922 actual=[114212, 102352, 4329]
token= uygulam encoded_token=Ġuygulam expected=106535 actual=[102983, 360, 309]
token= однако encoded_token=Ġоднако expected=116745 actual=[53419, 119553, 1482]
token= ulaş encoded_token=ĠulaÅŁ expected=104267 actual=[220, 5724, 7370]
token= jehož encoded_token=Ġjehož expected=118602 actual=[503, 2701, 100613]
token= özellikle encoded_token=Ġözellikle expected=109252 actual=[17372, 98329, 1609, 273]
token= jeho encoded_token=Ġjeho expected=101503 actual=[503, 2701, 78]
token= zákona encoded_token=Ġzákona expected=108998 actual=[1167, 72542, 6863]
token= увагу encoded_token=ĠÑĥвагÑĥ expected=122771 actual=[104852, 34493, 3865]
token= tématu encoded_token=Ġtématu expected=112487 actual=[47289, 8637, 84]
token= sıcak encoded_token=Ġsıcak expected=112915 actual=[274, 74383, 587]
token= rovněž encoded_token=ĠrovnÄĽÅ¾ expected=120242 actual=[938, 29238, 124830]
token=átní encoded_token=átnÃŃ expected=115721 actual=[1995, 1546, 2483]
token= располаг encoded_token=ĠÑĢаÑģполаг expected=127667 actual=[78876, 102500, 34493]
token=илакти encoded_token=илакÑĤи expected=122746 actual=[64292, 16248, 23311]
token=larından encoded_token=larından expected=105046 actual=[46937, 303, 276]
token= merkez encoded_token=Ġmerkez expected=107453 actual=[4809, 441, 89]
token= smlouvy encoded_token=Ġsmlouvy expected=116182 actual=[274, 1029, 283, 14029]
token= يناير encoded_token=ĠÙĬÙĨاÙĬر expected=116676 actual=[74374, 101537, 94643]
token= آینده encoded_token=ĠØ¢ÛĮÙĨدÙĩ expected=123831 actual=[100281, 83355, 92435]
token= etmektedir encoded_token=Ġetmektedir expected=120269 actual=[105805, 6702, 404]
token= Merkezi encoded_token=ĠMerkezi expected=115723 actual=[8930, 441, 8510]
token= görüntü encoded_token=Ġgörüntü expected=115385 actual=[58804, 2448, 406, 2448]
token=데이트 encoded_token=ëį°ìĿ´íĬ¸ expected=116494 actual=[167, 47318, 29726]
token= Cumhuriyeti encoded_token=ĠCumhuriyeti expected=121215 actual=[32678, 71, 6198, 47492, 72]
token= girdi encoded_token=Ġgirdi expected=120546 actual=[342, 2668, 72]
token=атися encoded_token=аÑĤиÑģÑı expected=106710 actual=[8131, 13810, 4329]
token= odstran encoded_token=Ġodstran expected=125015 actual=[11018, 496, 276]
token= geliş encoded_token=ĠgeliÅŁ expected=105969 actual=[342, 12574, 7370]
token= yıllar encoded_token=Ġyıllar expected=122956 actual=[122231, 657, 277]
token=ıldığı encoded_token=ıldıģı expected=111067 actual=[3862, 509, 75437]
token= липня encoded_token=ĠлипнÑı expected=120015 actual=[26539, 48445, 40251]
token= الکترون encoded_token=ĠاÙĦکترÙĪÙĨ expected=120542 actual=[17607, 102001, 86096, 12061]
token= baktı encoded_token=Ġbaktı expected=123844 actual=[293, 10114, 3862]
token= зустрі encoded_token=ĠзÑĥÑģÑĤÑĢÑĸ expected=122047 actual=[11122, 3865, 51745, 27385]
token= hlavně encoded_token=ĠhlavnÄĽ expected=119973 actual=[50686, 31697, 22161]
token= اروپا encoded_token=ĠارÙĪپا expected=124891 actual=[13258, 86096, 68483, 5821]
token= žádné encoded_token=Ġžádné expected=116312 actual=[32063, 1995, 17851, 978]
token= ihtiyaç encoded_token=Ġihtiyaç expected=110958 actual=[104112, 80574, 3209]
token= виступ encoded_token=ĠвиÑģÑĤÑĥп expected=116459 actual=[5927, 35111, 49907]
token= özellik encoded_token=Ġözellik expected=108501 actual=[17372, 98329, 1609]
token= poměrně encoded_token=ĠpomÄĽrnÄĽ expected=126829 actual=[30584, 22161, 124914]
token= благодаря encoded_token=ĠблагодаÑĢÑı expected=124779 actual=[117934, 105829, 4329]
token= электр encoded_token=ĠÑįлекÑĤÑĢ expected=120584 actual=[21599, 3114, 30656, 2233]
token= ovliv encoded_token=Ġovliv expected=120232 actual=[297, 14878, 344]
token= практи encoded_token=ĠпÑĢакÑĤи expected=104912 actual=[5173, 123282, 23311]
token=dıktan encoded_token=dıktan expected=127711 actual=[100376, 5964, 276]
token= považ encoded_token=Ġpovaž expected=113774 actual=[281, 12949, 12453]
token=İTESİ encoded_token=Ä°TESÄ° expected=122549 actual=[48880, 29584, 48880]
token= ситуации encoded_token=ĠÑģиÑĤÑĥаÑĨии expected=125925 actual=[5524, 102421, 54686]
token= diệt encoded_token=Ġdiá»ĩt expected=124723 actual=[1891, 26298, 83]
token= kvůli encoded_token=Ġkvůli expected=111620 actual=[16574, 52353, 747]
token= alespoň encoded_token=ĠalespoÅĪ expected=121678 actual=[264, 645, 119451]
token= gerekmektedir encoded_token=Ġgerekmektedir expected=118060 actual=[67387, 16400, 1247, 6702, 404]
token= olmadığını encoded_token=Ġolmadıģını expected=117426 actual=[8492, 20920, 93687, 38511]
token=üslü encoded_token=üslü expected=112803 actual=[2448, 3306, 2448]
token= bilgisayar encoded_token=Ġbilgisayar expected=117468 actual=[20934, 52198, 76246]
token= Tiểu encoded_token=ĠTiá»ĥu expected=127349 actual=[23126, 34550, 84]
token= ایران encoded_token=ĠاÛĮراÙĨ expected=101848 actual=[13258, 90920, 40523]
token=acaktır encoded_token=acaktır expected=109744 actual=[582, 10114, 30333]
token= každé encoded_token=Ġkaždé expected=125465 actual=[16909, 12453, 67, 978]
token= činnost encoded_token=ĠÄįinnost expected=122958 actual=[33902, 6258, 537]
token= mnohem encoded_token=Ġmnohem expected=120695 actual=[296, 2201, 30132]
token= kaynağı encoded_token=Ġkaynaģı expected=124457 actual=[37947, 3458, 44386]
token= yaptı encoded_token=Ġyaptı expected=111223 actual=[379, 2756, 3862]
token= çıktı encoded_token=Ġçıktı expected=113044 actual=[124711, 5964, 3862]
token= udělat encoded_token=ĠudÄĽlat expected=124595 actual=[28092, 22161, 5641]
token= předmět encoded_token=ĠpÅĻedmÄĽt expected=113990 actual=[59406, 90625, 100476]
token= Suriye encoded_token=ĠSuriye expected=124016 actual=[328, 6198, 9188]
token= электрон encoded_token=ĠÑįлекÑĤÑĢон expected=127472 actual=[21599, 3114, 30656, 103710]
token= культуры encoded_token=ĠкÑĥлÑĮÑĤÑĥÑĢÑĭ expected=126332 actual=[7820, 66005, 116584]
token= Václav encoded_token=ĠVáclav expected=123677 actual=[650, 1995, 566, 402]
token= günlük encoded_token=Ġgünlük expected=115452 actual=[55008, 16017, 40957]
token= yapılmış encoded_token=ĠyapılmÄ±ÅŁ expected=123965 actual=[85800, 18244, 33573]
token= культур encoded_token=ĠкÑĥлÑĮÑĤÑĥÑĢ expected=114750 actual=[7820, 66005, 56515]
token= 스타 encoded_token=ĠìĬ¤íĥĢ expected=114833 actual=[80307, 45780, 225, 222]
token= kiệm encoded_token=Ġkiá»ĩm expected=114148 actual=[20548, 26298, 76]
token= اقدام encoded_token=ĠاÙĤداÙħ expected=112556 actual=[13258, 101589, 50488]
token= літерату encoded_token=ĠлÑĸÑĤеÑĢаÑĤÑĥ expected=113190 actual=[102777, 58060, 103457]
token= Evropské encoded_token=ĠEvropské expected=127018 actual=[10641, 3787, 103770]
token= güneş encoded_token=ĠgüneÅŁ expected=119931 actual=[55008, 818, 7370]
token= vzděl encoded_token=ĠvzdÄĽl expected=110525 actual=[348, 49755, 100599]
token=átky encoded_token=átky expected=108653 actual=[1995, 6375, 88]
token= безпеки encoded_token=Ġбезпеки expected=122889 actual=[92457, 8164, 15298, 1840]
token= přibliž encoded_token=ĠpÅĻibliž expected=124069 actual=[59406, 69713, 12453]
token= přičemž encoded_token=ĠpÅĻiÄįemž expected=127371 actual=[59406, 84195, 336, 12453]
token= güncel encoded_token=Ġgüncel expected=115705 actual=[55008, 1031, 301]
token= Tarihi encoded_token=ĠTarihi expected=122188 actual=[350, 2850, 6151]
token= giriş encoded_token=ĠgiriÅŁ expected=109179 actual=[342, 21336, 7370]
token= характер encoded_token=ĠÑħаÑĢакÑĤеÑĢ expected=105670 actual=[45658, 1506, 123282, 58060]
token=:.:.:.:.: encoded_token=:.:.:.:.: expected=118496 actual=[17406, 17406, 108601]
token=átku encoded_token=átku expected=110150 actual=[1995, 6375, 84]
token=larındaki encoded_token=larındaki expected=125808 actual=[46937, 303, 14966]
token=ecektir encoded_token=ecektir expected=107572 actual=[75507, 5964, 404]
token= článek encoded_token=ĠÄįlánek expected=126539 actual=[33902, 101068, 48045]
token= Tiếng encoded_token=ĠTiếng expected=124942 actual=[23126, 27160, 983]
token= nhiệm encoded_token=Ġnhiá»ĩm expected=103562 actual=[308, 6151, 26298, 76]
token= виробництва encoded_token=ĠвиÑĢобниÑĨÑĤва expected=120393 actual=[5927, 78746, 14082, 109468]
token= miễn encoded_token=Ġmiá»ħn expected=113658 actual=[9686, 5080, 101792]
token= důležité encoded_token=Ġdůležité expected=122339 actual=[103440, 273, 12453, 13109]
token=.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.: encoded_token=.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.: expected=125437 actual=[124263, 17406, 17406, 17406, 17406, 17406, 17406, 17406, 17406, 17406, 17406, 108601]
token= заболеваний encoded_token=Ġзаболеваний expected=123846 actual=[101447, 7975, 33742, 105560]
token=итися encoded_token=иÑĤиÑģÑı expected=107658 actual=[9542, 13810, 4329]
token= května encoded_token=ĠkvÄĽtna expected=116040 actual=[16574, 22161, 1546, 64]
token= تلاش encoded_token=ĠتÙĦاش expected=118062 actual=[40534, 82070, 33890]
token= státní encoded_token=ĠstátnÃŃ expected=119946 actual=[357, 1995, 1546, 2483]
token= vztah encoded_token=Ġvztah expected=110600 actual=[348, 11949, 1494]
token= seviy encoded_token=Ġseviy expected=110410 actual=[513, 10176, 88]
token= používá encoded_token=ĠpoužÃŃvá expected=121356 actual=[17258, 12453, 59211, 1995]
token= üçüncü encoded_token=Ġüçüncü expected=120316 actual=[104610, 2448, 1031, 2448]
token=cılık encoded_token=cılık expected=116171 actual=[66, 53271, 74]
token= Üniversitesi encoded_token=ĠÃľniversitesi expected=107811 actual=[105766, 1986, 116493]
token= أبريل encoded_token=ĠأبرÙĬÙĦ expected=118513 actual=[64515, 84927, 96298]
token= biệt encoded_token=Ġbiá»ĩt expected=103147 actual=[6160, 26298, 83]
token= структур encoded_token=ĠÑģÑĤÑĢÑĥкÑĤÑĥÑĢ expected=126152 actual=[18868, 64478, 110584]
token=тися encoded_token=ÑĤиÑģÑı expected=120592 actual=[1830, 13810, 4329]
token= İnsan encoded_token=ĠÄ°nsan expected=113056 actual=[39521, 4511, 276]
token= zvlá encoded_token=Ġzvlá expected=111994 actual=[1167, 14878, 1995]
token= soubor encoded_token=Ġsoubor expected=127539 actual=[274, 5599, 269]
token= yaptık encoded_token=Ġyaptık expected=117592 actual=[379, 2756, 41990]
token=ımda encoded_token=ımda expected=117011 actual=[3862, 2329, 64]
token= zpráva encoded_token=Ġzpráva expected=125058 actual=[1167, 652, 1995, 6723]
token= присутств encoded_token=ĠпÑĢиÑģÑĥÑĤÑģÑĤв expected=119465 actual=[108490, 38521, 20812]
token= Každ encoded_token=ĠKažd expected=123584 actual=[23109, 12453, 67]
token= genellikle encoded_token=Ġgenellikle expected=115697 actual=[4173, 616, 1609, 273]
token= tức encoded_token=Ġtức expected=110939 actual=[47600, 102, 66]
token= yoğun encoded_token=ĠyoÄŁun expected=112806 actual=[30596, 11257, 359]
token= звичай encoded_token=ĠзвиÑĩай expected=118100 actual=[102098, 34082, 19039]
token= vurgu encoded_token=Ġvurgu expected=123954 actual=[348, 5673, 84]
token= söylem encoded_token=Ġsöylem expected=113065 actual=[51387, 982, 76]
token= умень encoded_token=ĠÑĥменÑĮ expected=116055 actual=[14257, 38972, 4929]
token= meydana encoded_token=Ġmeydana expected=114518 actual=[757, 41221, 3444]
token= měsíců encoded_token=ĠmÄĽsÃŃců expected=123971 actual=[100729, 82, 30515, 52353]
token= mềm encoded_token=Ġmá»ģm expected=107967 actual=[296, 41038, 76]
token= kurtul encoded_token=Ġkurtul expected=122690 actual=[597, 5757, 360]
token= aylık encoded_token=Ġaylık expected=123634 actual=[264, 4010, 41990]
token= принимать encoded_token=ĠпÑĢинимаÑĤÑĮ expected=127246 actual=[107279, 16494, 18482]
token=talya encoded_token=talya expected=112728 actual=[83, 5893, 64]
token=.::.:: encoded_token=.::.:: expected=114282 actual=[13, 58204, 487]
token= вироб encoded_token=ĠвиÑĢоб expected=105983 actual=[5927, 78746, 14082]
token= zpracování encoded_token=ĠzpracovánÃŃ expected=122639 actual=[1167, 652, 582, 100956]
token= olacaktır encoded_token=Ġolacaktır expected=112623 actual=[8492, 582, 10114, 30333]
token= baskı encoded_token=Ġbaskı expected=124205 actual=[293, 1091, 3862]
token=кадем encoded_token=кадем expected=112063 actual=[4898, 23680, 10298]
token=되었습니다 encoded_token=ëIJĺìĹĪìĬµëĭĪëĭ¤ expected=124771 actual=[65219, 13879, 90463]
token= zákaz encoded_token=Ġzákaz expected=113646 actual=[1167, 72542, 1394]
token= вияв encoded_token=ĠвиÑıв expected=114040 actual=[5927, 40231, 5591]
token= všechno encoded_token=ĠvÅ¡echno expected=112196 actual=[109730, 2174, 78]
token= индивиду encoded_token=ĠиндивидÑĥ expected=120375 actual=[42078, 111506, 48262, 3865]
token= jednoduch encoded_token=Ġjednoduch expected=121004 actual=[18806, 35666, 1412]
token=ılmaz encoded_token=ılmaz expected=117857 actual=[3862, 18244, 1394]
token= zásob encoded_token=Ġzásob expected=127493 actual=[1167, 7206, 677]
token= например encoded_token=ĠнапÑĢимеÑĢ expected=111797 actual=[91105, 2233, 81448]
token=áhnout encoded_token=áhnout expected=122032 actual=[1995, 25105, 412]
token= otev encoded_token=Ġotev expected=107395 actual=[297, 668, 85]
token= döndü encoded_token=Ġdöndü expected=127347 actual=[107766, 303, 2448]
token= türlü encoded_token=Ġtürlü expected=113197 actual=[259, 2448, 2438, 2448]
token= kiểu encoded_token=Ġkiá»ĥu expected=113229 actual=[20548, 34550, 84]
token= Yönet encoded_token=ĠYönet expected=111568 actual=[816, 3029, 4816]
token=ımlı encoded_token=ımlı expected=127367 actual=[3862, 1029, 3862]
token=ーカー encoded_token=ãĥ¼ãĤ«ãĥ¼ expected=114533 actual=[38248, 104, 11972]
token= düşünc encoded_token=ĠdÃ¼ÅŁÃ¼nc expected=121671 actual=[101177, 2448, 1031]
token= údaj encoded_token=Ġúdaj expected=111045 actual=[18448, 67, 1662]
token= önemlidir encoded_token=Ġönemlidir expected=123636 actual=[102075, 62306, 404]
token=ючись encoded_token=ÑİÑĩиÑģÑĮ expected=115216 actual=[12182, 113001, 4929]
token= söyley encoded_token=Ġsöyley expected=116452 actual=[51387, 982, 88]
token=уватися encoded_token=ÑĥваÑĤиÑģÑı expected=112162 actual=[100396, 8131, 13810, 4329]
token= meziná encoded_token=Ġmeziná expected=118377 actual=[757, 58197, 1995]
token= parçası encoded_token=Ġparçası expected=122601 actual=[1370, 46378, 3862]
token= виріш encoded_token=ĠвиÑĢÑĸÑĪ expected=113004 actual=[5927, 78746, 102727]
token= адміністратив encoded_token=ĠадмÑĸнÑĸÑģÑĤÑĢаÑĤив expected=126626 actual=[103186, 108806, 51745, 104758]
token= наличи encoded_token=ĠналиÑĩи expected=115456 actual=[13373, 91216, 1840]
token= федераль encoded_token=ĠÑĦедеÑĢалÑĮ expected=123368 actual=[18034, 13404, 111458]
token= میدان encoded_token=ĠÙħÛĮداÙĨ expected=127100 actual=[24252, 66143, 40523]
token= возду encoded_token=ĠвоздÑĥ expected=107562 actual=[5927, 35387, 3865]
token= niệm encoded_token=Ġniá»ĩm expected=113632 actual=[13080, 26298, 76]
token= geliştir encoded_token=ĠgeliÅŁtir expected=107286 actual=[342, 12574, 99439]
token= ustanovení encoded_token=ĠustanovenÃŃ expected=120536 actual=[577, 24986, 16651, 2483]
token=alardan encoded_token=alardan expected=123997 actual=[278, 569, 276]
token= اطلاع encoded_token=ĠاطÙĦاع expected=124833 actual=[102992, 82070, 24102]
token= všechny encoded_token=ĠvÅ¡echny expected=106787 actual=[109730, 2174, 88]
token= değiş encoded_token=ĠdeÄŁiÅŁ expected=103425 actual=[409, 44907, 7370]
token= fotoğraf encoded_token=ĠfotoÄŁraf expected=117072 actual=[26656, 11257, 15817]
token= název encoded_token=Ġnázev expected=122846 actual=[100771, 3059, 85]
token= toplantı encoded_token=Ġtoplantı expected=115994 actual=[311, 21494, 3862]
token= Например encoded_token=ĠÐĿапÑĢимеÑĢ expected=122711 actual=[35889, 19619, 2233, 81448]
token= uvědom encoded_token=ĠuvÄĽdom expected=125654 actual=[30763, 22161, 5717]
token=.:.:.:.:.:.:.:.: encoded_token=.:.:.:.:.:.:.:.: expected=105356 actual=[124263, 17406, 17406, 108601]
token= mümkün encoded_token=Ġmümkün expected=107737 actual=[59385, 25457, 16461]
token= připoj encoded_token=ĠpÅĻipoj expected=125491 actual=[59406, 7088, 73]
token= академ encoded_token=Ġакадем expected=117319 actual=[86430, 23680, 10298]
token= taşıy encoded_token=ĠtaÅŁÄ±y expected=119003 actual=[9637, 78413, 88]
token= stejně encoded_token=ĠstejnÄĽ expected=110120 actual=[4179, 94908, 22161]
token= 업데이트 encoded_token=ĠìĹħëį°ìĿ´íĬ¸ expected=120423 actual=[107022, 167, 47318, 29726]
token= yardımcı encoded_token=Ġyardımcı expected=110324 actual=[20085, 3862, 13183, 3862]
token= nebezpeč encoded_token=ĠnebezpeÄį expected=125786 actual=[841, 1395, 89, 105629]
token=еристи encoded_token=еÑĢиÑģÑĤи expected=110043 actual=[7753, 35111, 1840]
token= yaptır encoded_token=Ġyaptır expected=118761 actual=[379, 2756, 30333]
token= характериз encoded_token=ĠÑħаÑĢакÑĤеÑĢиз expected=120702 actual=[45658, 1506, 123282, 58060, 41873]
token= století encoded_token=ĠstoletÃŃ expected=109109 actual=[43132, 1169, 2483]
token= değiştir encoded_token=ĠdeÄŁiÅŁtir expected=108303 actual=[409, 44907, 99439]
token= yıldız encoded_token=Ġyıldız expected=126575 actual=[122231, 509, 48980]
token=üslüman encoded_token=üslüman expected=114091 actual=[2448, 3306, 2448, 1543]
token= sıkıntı encoded_token=Ġsıkıntı expected=120121 actual=[104223, 3862, 406, 3862]
token=mektedir encoded_token=mektedir expected=102909 actual=[74853, 6702, 404]
token=lásil encoded_token=lásil expected=126647 actual=[75, 7206, 321]
token= vyrá encoded_token=Ġvyrá expected=127531 actual=[348, 11160, 1995]
token= натураль encoded_token=ĠнаÑĤÑĥÑĢалÑĮ expected=126285 actual=[111640, 56515, 39127]
token= otevř encoded_token=ĠotevÅĻ expected=112206 actual=[297, 668, 85, 29432]
token= зависит encoded_token=ĠзавиÑģиÑĤ expected=115635 actual=[93323, 13810, 9542]
token= Ayrıca encoded_token=ĠAyrıca expected=107716 actual=[362, 11160, 3862, 936]
token= sonrasında encoded_token=Ġsonrasında expected=118452 actual=[4538, 13075, 39797]
token= sağlıklı encoded_token=ĠsaÄŁlıklı expected=122012 actual=[85939, 26693, 10784, 3862]
token=lamaktadır encoded_token=lamaktadır expected=127577 actual=[24705, 10114, 101252]
token= дозвол encoded_token=Ġдозвол expected=120325 actual=[7952, 49855, 7975]
token=ıldığında encoded_token=ıldıģında expected=127896 actual=[3862, 509, 93687, 39797]
token= yıllarda encoded_token=Ġyıllarda expected=122359 actual=[122231, 657, 72626]
token= краще encoded_token=ĠкÑĢаÑīе expected=120005 actual=[7820, 52866, 1532]
token= використання encoded_token=ĠвикоÑĢиÑģÑĤаннÑı expected=113315 actual=[107038, 9239, 35111, 100386]
token= thỏa encoded_token=Ġthá»ıa expected=121167 actual=[270, 86242, 64]
token= практически encoded_token=ĠпÑĢакÑĤиÑĩеÑģки expected=117794 actual=[5173, 123282, 23311, 109859]
token= .**************
encoded_token=Ġ.**************Ċ expected=123046 actual=[662, 601, 65171]
token= hlavní encoded_token=ĠhlavnÃŃ expected=113095 actual=[50686, 31697, 2483]
token= údaje encoded_token=Ġúdaje expected=114324 actual=[18448, 67, 11305]
token= Yönetim encoded_token=ĠYönetim expected=119362 actual=[816, 3029, 4816, 318]
token=ávací encoded_token=ávacÃŃ expected=118260 actual=[1995, 54803, 2483]
token= zprac encoded_token=Ġzprac expected=110655 actual=[1167, 652, 582]
token= zdroj encoded_token=Ġzdroj expected=115378 actual=[1167, 11513, 73]
token= obsahuje encoded_token=Ġobsahuje expected=121717 actual=[7600, 27793, 3841]
token= Coğraf encoded_token=ĠCoÄŁraf expected=127877 actual=[3623, 11257, 15817]
token=acılık encoded_token=acılık expected=113983 actual=[582, 53271, 74]
token= مدينة encoded_token=ĠÙħدÙĬÙĨØ© expected=117188 actual=[24252, 80920, 101788]
token= Belediyesi encoded_token=ĠBelediyesi expected=124814 actual=[2893, 104181, 9891, 72]
token= chtě encoded_token=ĠchtÄĽ expected=109702 actual=[272, 427, 22161]
token= заяв encoded_token=ĠзаÑıв expected=108123 actual=[11122, 36497, 5591]
token= riêng encoded_token=Ġriêng expected=107000 actual=[25160, 5615, 983]
token= 오후 encoded_token=Ġìĺ¤íĽĦ expected=124467 actual=[39623, 45780, 75309]
token=토토 encoded_token=íĨłíĨł expected=119927 actual=[100757, 58260, 228, 254]
token= podmínek encoded_token=ĠpodmÃŃnek expected=125711 actual=[7661, 111737, 48045]
token= зависимости encoded_token=ĠзавиÑģимоÑģÑĤи expected=114885 actual=[93323, 13810, 116776]
token= dítě encoded_token=ĠdÃŃtÄĽ expected=111877 actual=[294, 16485, 22161]
token= ایرانی encoded_token=ĠاÛĮراÙĨÛĮ expected=113399 actual=[13258, 90920, 100819]
token= ayrıntılı encoded_token=Ġayrıntılı expected=125971 actual=[110221, 406, 53271]
token= olursa encoded_token=Ġolursa expected=119830 actual=[8492, 1759, 64]
token= nhiệt encoded_token=Ġnhiá»ĩt expected=107421 actual=[308, 6151, 26298, 83]
token= اکتبر encoded_token=Ġاکتبر expected=127043 actual=[13258, 102001, 84927]
token=muştur encoded_token=muÅŁtur expected=113050 actual=[15479, 48497, 324]
token= olmaktadır encoded_token=Ġolmaktadır expected=127453 actual=[100703, 10114, 101252]
token= ÜNİVERSİTESİ encoded_token=ĠÃľNÄ°VERSÄ°TESÄ° expected=123981 actual=[123073, 48880, 29584, 48880]
token= gerektiğini encoded_token=ĠgerektiÄŁini expected=119850 actual=[67387, 5964, 72, 101588]
token= özelliği encoded_token=ĠözelliÄŁi expected=127172 actual=[101213, 21148, 44907]
token= руковод encoded_token=ĠÑĢÑĥковод expected=114456 actual=[18600, 104484, 9706]
token=ární encoded_token=árnÃŃ expected=107818 actual=[1995, 36722, 2483]
token= Kültür encoded_token=ĠKültür expected=123508 actual=[70352, 4937, 5297]
token= diễn encoded_token=Ġdiá»ħn expected=106371 actual=[1891, 5080, 101792]
token= 스포츠 encoded_token=ĠìĬ¤íı¬ì¸ł expected=126673 actual=[80307, 45780, 237, 105, 104554]
token= глаза encoded_token=Ġглаза expected=111043 actual=[25756, 3114, 109424]
token= belirlen encoded_token=Ġbelirlen expected=126445 actual=[1689, 2881, 268]
token=ıntı encoded_token=ıntı expected=106216 actual=[3862, 406, 3862]
token= hợp encoded_token=Ġhợp expected=100827 actual=[305, 85400, 79]
token= vzdál encoded_token=Ġvzdál expected=115944 actual=[348, 49755, 19540]
token= teklif encoded_token=Ġteklif expected=120911 actual=[1028, 10784, 333]
token= Федерации encoded_token=ĠФедеÑĢаÑĨии expected=111112 actual=[126723, 7753, 54686]
token= milyar encoded_token=Ġmilyar expected=117001 actual=[296, 1570, 277]
token= každ encoded_token=Ġkažd expected=103870 actual=[16909, 12453, 67]
token= uveden encoded_token=Ġuveden expected=108512 actual=[577, 2111, 268]
token= zásad encoded_token=Ġzásad expected=123705 actual=[1167, 7206, 329]
token= Olomou encoded_token=ĠOlomou expected=121828 actual=[12225, 316, 283]
token= назва encoded_token=Ġназва expected=119618 actual=[6850, 57059, 1506]
token= культу encoded_token=ĠкÑĥлÑĮÑĤÑĥ expected=104937 actual=[7820, 66005, 3865]
token= качества encoded_token=ĠкаÑĩеÑģÑĤва expected=122689 actual=[7820, 50223, 103747]
token=илась encoded_token=илаÑģÑĮ expected=114692 actual=[64292, 18437, 4929]
token=lardı encoded_token=lardı expected=108457 actual=[75, 569, 3862]
token=中文字幕 encoded_token=ä¸ŃæĸĩåŃĹå¹ķ expected=126348 actual=[16325, 88435, 106633]
token= вихов encoded_token=ĠвиÑħов expected=117929 actual=[5927, 67745, 6856]
token= kaldı encoded_token=Ġkaldı expected=115060 actual=[597, 4852, 3862]
token= duyệt encoded_token=Ġduyá»ĩt expected=119799 actual=[102082, 26298, 83]
token= ngoài encoded_token=ĠngoÃłi expected=102507 actual=[308, 3427, 53651]
token= itibaren encoded_token=Ġitibaren expected=113490 actual=[433, 581, 9329]
token= Arthropoda encoded_token=ĠArthropoda expected=112328 actual=[1676, 98643, 14320]
token= yapıldı encoded_token=Ġyapıldı expected=122941 actual=[85800, 509, 3862]
token= ремонт encoded_token=ĠÑĢемонÑĤ expected=122847 actual=[18600, 10298, 97341]
token= جديدة encoded_token=ĠجدÙĬدة expected=119846 actual=[83268, 80920, 102539]
token= sahiptir encoded_token=Ġsahiptir expected=115757 actual=[829, 6151, 418, 404]
token= hukuk encoded_token=Ġhukuk expected=119422 actual=[305, 3178, 3178]
token= Erdoğan encoded_token=ĠErdoÄŁan expected=115026 actual=[9939, 3055, 124088]
token=lıklar encoded_token=lıklar expected=117691 actual=[26693, 10784, 277]
token= характеристики encoded_token=ĠÑħаÑĢакÑĤеÑĢиÑģÑĤики expected=125355 actual=[45658, 1506, 123282, 58060, 35111, 105326]
token= культури encoded_token=ĠкÑĥлÑĮÑĤÑĥÑĢи expected=118659 actual=[7820, 66005, 117982]
token= değildir encoded_token=ĠdeÄŁildir expected=108644 actual=[62087, 699, 404]
token= зазнач encoded_token=ĠзазнаÑĩ expected=117098 actual=[11122, 19188, 31281]
token= odkazy encoded_token=Ġodkazy expected=111096 actual=[297, 7737, 13933]
token= şeklinde encoded_token=ĠÅŁeklinde expected=114410 actual=[94395, 3817, 451]
token= mevcut encoded_token=Ġmevcut expected=110387 actual=[757, 7513, 332]
token= الدين encoded_token=ĠاÙĦدÙĬÙĨ expected=115451 actual=[17607, 80920, 12061]
token= диза encoded_token=Ġдиза expected=120581 actual=[7952, 41873, 1506]
token= الحياة encoded_token=ĠاÙĦØŃÙĬاة expected=112224 actual=[100688, 101160, 26957]
token= заним encoded_token=Ġзаним expected=124114 actual=[11122, 14332, 6578]
token= miệng encoded_token=Ġmiá»ĩng expected=122631 actual=[9686, 26298, 983]
token= трохи encoded_token=ĠÑĤÑĢоÑħи expected=120928 actual=[86995, 101897, 1840]
token= liệt encoded_token=Ġliá»ĩt expected=125958 actual=[908, 26298, 83]
token= лаборатор encoded_token=ĠлабоÑĢаÑĤоÑĢ expected=127014 actual=[26539, 38722, 9239, 58406]
token= визначення encoded_token=ĠвизнаÑĩеннÑı expected=125410 actual=[5927, 41873, 31281, 100359]
token= oyuncu encoded_token=Ġoyuncu expected=119750 actual=[70448, 1371, 84]
token= sürekli encoded_token=Ġsürekli expected=114117 actual=[60839, 42961, 747]
token= використов encoded_token=ĠвикоÑĢиÑģÑĤов expected=107197 actual=[107038, 9239, 35111, 6856]
token= belirtilen encoded_token=Ġbelirtilen expected=120909 actual=[101664, 1678, 268]
token=lıkları encoded_token=lıkları expected=120454 actual=[26693, 10784, 68228]
token= فناوری encoded_token=ĠÙģÙĨاÙĪرÛĮ expected=122228 actual=[46677, 101537, 102270]
token=lardan encoded_token=lardan expected=103084 actual=[75, 569, 276]
token= thiểu encoded_token=Ġthiá»ĥu expected=119293 actual=[62428, 34550, 84]
token= gelişim encoded_token=ĠgeliÅŁim expected=127504 actual=[342, 12574, 7370, 318]
token= nhiễ encoded_token=Ġnhiá»ħ expected=112969 actual=[308, 6151, 103490]
token= ředitel encoded_token=ĠÅĻeditel expected=120621 actual=[101259, 3671, 301]
token= méně encoded_token=ĠménÄĽ expected=118476 actual=[296, 10610, 22161]
failed 588 skipped 1352
Taking "Ġsoubor" as an example, the decoded string is " soubor". Taking the token IDs for the individual bytes and applying merge rules, the encoded IDs are [274, 5599, 269] ("Ġs", "oub", "or") instead of the single token [127539]. There is a merge pair ("Ġsou", "bor"), but it never gets used because of the order in which merges are applied according to their ranks. The sequence of merges is:
[220, 82, 78, 84, 65, 78, 81]
[220, 82, 78, 84, 65, 269]
[274, 78, 84, 65, 269]
[274, 283, 65, 269]
[274, 5599, 269]