Add aditional lang folder for skill loader to look into

[andlo]

Some skills are missing translations or the translation on poodle isnt yet merged. At the same time when adding translation to poodle it is nearly imposible to test it right away. What I am proposing is adding a aditional language folder outside of skills where aditional language files can be plased. That would make it posible to translate localy or fetch translation from poodle and use it right away.

My proposal is adding a setting to mycroft.comf for aditional_language=/opt/mycroft/aditional_language/

Then when skill loader looks for lang files it looks first in skillfolder and if none exists it looks at aditional_lanuage for skill folder and lang files there.

Then a skil can be made that fetches langfiles from poodle and place them in aditional_language folder. Or a skill that auto translate and place files in there as wel or a combo of the two.

In mycroft-skills.py the functions that reads lang files could be changed from

   def load_vocab_files(self, root_directory):
        """ Load vocab files found under root_directory.

        Arguments:
            root_directory (str): root folder to use when loading files
        """
        keywords = []
        vocab_dir = join(root_directory, 'vocab', self.lang)
        locale_dir = join(root_directory, 'locale', self.lang)
        if exists(vocab_dir):
            keywords = load_vocabulary(vocab_dir, self.skill_id)
        elif exists(locale_dir):
            keywords = load_vocabulary(locale_dir, self.skill_id)
        else:
            LOG.debug('No vocab loaded')

        # For each found intent register the default along with any aliases
        for vocab_type in keywords:
            for line in keywords[vocab_type]:
                entity = line[0]
                aliases = line[1:]
                self.intent_service.register_adapt_keyword(vocab_type,
                                                           entity,
                                                           aliases)

    def load_regex_files(self, root_directory):
        """ Load regex files found under the skill directory.

        Arguments:
            root_directory (str): root folder to use when loading files
        """
        regexes = []
        regex_dir = join(root_directory, 'regex', self.lang)
        locale_dir = join(root_directory, 'locale', self.lang)
        if exists(regex_dir):
            regexes = load_regex(regex_dir, self.skill_id)
        elif exists(locale_dir):
            regexes = load_regex(locale_dir, self.skill_id)

        for regex in regexes:
            self.intent_service.register_adapt_regex(regex)

to something like this

   def load_vocab_files(self, root_directory):
        """ Load vocab files found under root_directory.

        Arguments:
            root_directory (str): root folder to use when loading files
        """
        keywords = []
        vocab_dir = join(root_directory, 'vocab', self.lang)
        locale_dir = join(root_directory, 'locale', self.lang)
        if exists(vocab_dir):
            keywords = load_vocabulary(vocab_dir, self.skill_id)
        elif exists(locale_dir):
            keywords = load_vocabulary(locale_dir, self.skill_id)
        else:
            LOG.debug('No vocab loaded')

        # For each found intent register the default along with any aliases
        for vocab_type in keywords:
            for line in keywords[vocab_type]:
                entity = line[0]
                aliases = line[1:]
                self.intent_service.register_adapt_keyword(vocab_type,
                                                           entity,
                                                           aliases)

    def load_regex_files(self, root_directory):
        """ Load regex files found under the skill directory.

        Arguments:
            root_directory (str): root folder to use when loading files
        """
        regexes = []
        regex_dir = join(root_directory, 'regex', self.lang)
        locale_dir = join(root_directory, 'locale', self.lang)
        aditional_dir = join(settings.get("aditional_language", self.lang)
        if exists(regex_dir):
            regexes = load_regex(regex_dir, self.skill_id)
        elif exists(locale_dir):
            regexes = load_regex(locale_dir, self.skill_id)
        elif exists(aditional_dir):
            regexes = load_regex(locale_dir, self.skill_id)

        for regex in regexes:
            self.intent_service.register_adapt_regex(regex)

I have an autotranslate skill reddy but if it saves the lang files into a skill, the skill will not update later on while it has been changed. By having the files outside the skill the skill will update as usaly.

[forslund]

I think that's a great idea, we could also create a script to pull down translations from pootle to the additional_language folder if someone wants to test / verify their translations done there.

[andlo]

Yah that were what I think a skill could do. But but Where should the aditional lang folder exists and what should it be named ? Aditional_language or local_lang or what ? Should it be in ~/.mycroft/ /opt/mycroft/ or under a third place ?

And should each skill with aditional language be in folder like folder in skills or like folders in .mycroft/skills ?

And shoud the just be in a locale/lang folder or should it be like the skill where some of them has vocab, regex and dialog folders ?

I can make it as I think I have found the right place to add it and I previosly has made adings to mycroft.conf. But need som design directions.

Also it can test if files in lang folder matches en-us and if not add those files in the aditional language (either from poodle or from auto translation.

[andlo]

I found a way to pull down the .po files thanks to @gras64 and he alsp has a way to transform them to mycroft format files. so that part is more or les covered. I hassnt yet found a way to upload :( but that would als be great but not an issue for this proposal.

[forslund]

There is also a script in the mycroft-translate repo for creating the translated mycroft files.

If you want to be able to do it from a skill it may be best to use the ~/.mycroft/ folder as base since that is guaranteed to be there and writable no matter which user is running core. I would personally call the translation-folder translations

[andlo]

Then ~/.mycroft/translations it is Ill look into the mycroft-translate folder and use what I can from that :) and the setting in mycroft.con should be called ? translations_folder? translations_dir?

And shiuld there be a setting for what to use first - skill or translation (if both is present)? prefear_translations: true/false

I think a skill for pulling translations is the best way to go as it would make smallest impact into core and a flexibale way tat is open to other ways to get translations (

[forslund]

I agree that a skill is probably preferable.

Seems like the config is using _dir mostly so translations_dir is likely best prefere_translations sounds like a good name for the config :+1:

I also think we should listen to @krisgesling, if he has any preference since he is doing a lot of skill and translation work.

[andlo]

yep - As soon as posible Ill make some work on this and then we can see where it is going. @gras64 is working on a skill to pull down the translaions so I can do work on adding the translation folder and get that part to work. And then I can change my auto translate skill to use that folder as well.

I need to figure out how to get my fork of mycroft-core up to date and rebased so it isnt dooing strange stuff when I make a PR.

Then next step could be to synk the other way so translatios could upload to poodle.

[krisgesling]

I don't have much to add, just to say that I absolutely love this.

Thanks for taking the initiative again Andlo!

[denics]

In all my mycroft installation I override mycroft-core/skills/fallback-unknown.mycroftai/dialog/it-it/unknown.dialog to only have a short "non ho capito.". can we have something like .config/mycroft/skills/fallback-unknown.mycroftai/dialog/it-it/unknown.dialog that override the original file so we do not have to worry to lose everything?

having a .config/mycroft/skills/ folder may allow also, in the future, to override remote configuration ;)