Individual orthographies not detected by CharsetChecker

[arialcrime]

It might very well be that I am misunderstanding/misusing something, but I think there is something odd about hyperglot’s CharsetChecker when it comes to Serbian.

Somehow it does not detect support for the individual orthographies. It only comes back True when both Latin and Cyrillic is supported.

Should I be approaching this somehow differently? Is there a way for me to get the individual orthographies return True in this case?

Thanks for your help!

Here is some sample code to illustrate the issue I ran into: (disclaimer: hyperglot feeding itself is for demostration purposes only here)

from hyperglot.languages import Languages
from hyperglot.checker import CharsetChecker

tag = "srp" # different, but somewhat comparable issue also showed up with "uzn"

char_dict = dict()

for o in Languages()[tag]["orthographies"]:
    if not o["script"] in char_dict:
        char_dict[o["script"]] = ""
    # combine base, aux, and marks into one string to use in checking later
    char_dict[o["script"]] = " ".join([o["base"], o["auxiliary"], o["marks"]])

# go through scripts individually and check for support
for script, characters in char_dict.items():
    print(script.upper(), CharsetChecker(characters).supports_language(tag)) # check_all_orthographies=False does not change things here

# Combining all characters across scripts returns True
all_chars = " ".join([c for s, c in char_dict.items()])
print("ALL", CharsetChecker(all_chars).supports_language(tag))

Hyperglot version: 0.7.1

[kontur]

This is expected behavior, the logic being that both Cyrillic and Latin are indeed required to faithfully support Serbian (see wiki). It is my understanding that either script is valid and it is up to the publisher/media which they use. While Serbian can be written with either script, it is our interpretation that to claim support for Serbian as a whole, both scripts are required. There are only a few cases of such a requirement in the entire database, but where it is, this is intentional. Of course, feel free to debate this decision. :)

Technically, it's the preferred_as_group option, as documented in the README here:

preferred_as_group (optional, defaults to false) will combine all orthographies of this language. When used, a language is detected as supported only if all its orthographies with this attribute are supported. This is used for Serbian to require both Cyrillic-script and Latin-script orthographies to be supported and for Japanese to require Hiragana, Katakana, and Kanji orthographies to be supported.

And for Serbian in particular it is set here and here.

Does this make sense?