Grammar Exclusivity Manager

[LexiconCode]

Is your feature request related to a problem? Please describe. Credit goes to @davidlehub for initial creation of idea and implementation thus far.

Dragonfly grammar exclusivity allows grammars to be exclusive over normal grammars. Exclusive grammars are only recognized.

This allows for a few things.

• Exclusive Grammars can override dragons built in grammars.
• Hopefully makes creating different modes like spelling mode and such easier to implement.

Describe the solution you'd like A system to manage grammar exclusivity. To what extent an exact implementation yet to be determined.

Preferably implemented in casters Hooks and Events system for minimal impact on core code.

Describe alternatives you've considered The only alternative is to disable all of the rules and even then dragons built in rules interfere.

Summarized quotes and resources from davidlehub from gitter.

davidlehub

I putted Exclusivity Manager in 1 file. (for people to test) 1.Download this "init.py" from https://drive.google.com/open?id=1BD8JojzPaROke_N9ATApBt74QC3bTaUv 2. Put that file in Caster folder ("Documents\Caster")

davidlehub

Relate to "hooks", i suspect the "HookRunner" could be interesting. Maybe this could help: pictures

davidlehub

To make that system more useful. I had to: Make it works per app: Auto add "context app" rules to be exclusive, when foreground windows changed. Store/remember witch rules are been exclusives per app. Enable Dragon vocabularies when needed. etc.

[davidlehub]

thks

On Sat, Feb 1, 2020 at 4:39 PM LexiconCode [email protected] wrote:

Is your feature request related to a problem? Please describe. Credit goes to @davidlehub https://github.com/davidlehub for initial creation of idea and implementation thus far.

Dragonflies grammar exclusivity allows grammars to be exclusive over normal grammars. Exclusive grammars are only recognized.

This allows for a few things.

• Exclusive Grammars can override dragons built in grammars.
• Hopefully makes creating different modes like spelling mode and such easier to implement.

Describe the solution you'd like A system to manage grammar exclusivity. To what extent an exact implementation yet to be determined.

Preferably implemented in casters Hooks and Events system for minimal impact on core code.

Describe alternatives you've considered The only alternative is to disable all of the rules and even then dragons built in rules interfere.

Summarized quotes and resources from davidlehub from gitter.

davidlehub

I putted Exclusivity Manager in 1 file. (for people to test) 1.Download this "init.py" from https://drive.google.com/open?id=1BD8JojzPaROke_N9ATApBt74QC3bTaUv

1. Put that file in Caster folder ("Documents\Caster")

davidlehub

Relate to "hooks", i suspect the "HookRunner" could be interesting. Maybe this could help: pictures https://drive.google.com/open?id=14Crr7BhV5vwxJHYm08NmNEPuUNLuPgzE

davidlehub

To make that system more useful. I had to: Make it works per app: Auto add "context app" rules to be exclusive, when foreground windows changed. Store/remember witch rules are been exclusives per app. Enable Dragon vocabularies when needed. etc.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dictation-toolbox/Caster/issues/738?email_source=notifications&email_token=ALNO3O3Y4AFSWWK22UN7RSLRAXTYFA5CNFSM4KOVTIXKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IKLS2YQ, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALNO3O4IHM5YWGMUILC3SIDRAXTYFANCNFSM4KOVTIXA .

[davidlehub]

:) Also, maybe the videos? It could help peoples to have some idea how exclusiveness can be useful.

[LexiconCode]

@davidlehub That would be a good idea. You can even update your current post with the videos.

[davidlehub]

Here they are: https://www.youtube.com/playlist?list=PLQ6G9K9v8sUR44kvrit9pFoSBvGPKgM-7

Again, sorry for my English end the slowness. My firs time self recording explaining some things :D

[lettersandnumbersgithub]

I found the following in \Dragonfly\dragonfly\grammar\grammar_base.py Perhaps it could be used to enable/disable exclusive mode when switching between windows/applications?

    def enter_context(self):
        """
            Enter context callback.

            This method is called when a phrase-start has been
            detected.  It is only called if this grammar's
            context previously did not match but now does
            match positively.

        """

    def exit_context(self):
        """
            Exit context callback.

            This method is called when a phrase-start has been
            detected.  It is only called if this grammar's
            context previously did match but now doesn't
            match positively anymore.

        """

Also, If all Dragon commands are disabled when in exclusive mode, perhaps we should make a list of, and devise workarounds/pass-throughs for, any Dragon commands we wish to keep? Such as: "Go To Sleep" "Wake Up" (if also excluded) "Click [buttontext]" "Press [modifiers] [key]" If Mimic() won't work in exclusive mode?

[davidlehub]

I found the following in \Dragonfly\dragonfly\grammar\grammar_base.py Perhaps it could be used to enable/disable exclusive mode when switching between windows/applications?

    def enter_context(self):
        """
            Enter context callback.

            This method is called when a phrase-start has been
            detected.  It is only called if this grammar's
            context previously did not match but now does
            match positively.

        """

    def exit_context(self):
        """
            Exit context callback.

            This method is called when a phrase-start has been
            detected.  It is only called if this grammar's
            context previously did match but now doesn't
            match positively anymore.

        """

Also, If all Dragon commands are disabled when in exclusive mode, perhaps we should make a list of, and devise workarounds/pass-throughs for, any Dragon commands we wish to keep? Such as: "Go To Sleep" "Wake Up" (if also excluded) "Click " "Press " If Mimic() won't work in exclusive mode?

Relate to that, there is also:

    def _process_begin(self, executable, title, handle):
        """
            Start of phrase callback.

            *This usually is the method which should be overridden
            to give derived grammar classes custom behavior.*

            This method is called when the speech recognition
            engine detects that the user has begun to speak a
            phrase. This method is called by the
            ``Grammar.process_begin`` method only if this
            grammar's context matches positively.

            Arguments:
             - *executable* -- the full path to the module whose
               window is currently in the foreground.
             - *title* -- window title of the foreground window.
             - *handle* -- window handle to the foreground window.

        """

Example, this is what i did initially to detect >> foreground windows changed :

class grammExclusivenessCtrler_rule(MergeRule):
    # mapping = {
    # }
    
    def _process_begin(self):
        # ...
        
        activeWinHnd = win32gui.GetForegroundWindow()
        if activeWinHnd != gl.previousWinHnd:

            """ Forground windows changed, so:
            Make (or restore) a set of rules/grammars to be exclusive for the current forground window.
            """

            # ...
            
            gl.previousWinHnd = activeWinHnd

note the following: - That "_process_begin" function is called ONLY when the microphone start to dectecting some things (e.g., the user start to speak in to the microphone). So that means, it not detect "foreground window changed" as soon it happens. And that could cause a little latency if the process, of making the related rules/grammars to be exclusive, takes time...

(That’s why, since i don’t know how with python, i used an external tool, Autohotkey, to detect the "foreground window changed" EVENT, then tell Caster it happens. )

[davidlehub]

"Go To Sleep" "Wake Up" (if also excluded) "Click " "Press "

They are all excluded (all Dragon commands).

(Yeaa, our Dragon is not happy and turns a deaf ear, bcz we excluded him :-D )

As Solution, what i did:

• hmm, i think is more easy to explain with a screen view... i will post a video.

In short:

1. Store the "state" of the current exclusiveness.
2. Disable exclusiveness of all grammar. It makes Dragon commands available.
3. Do the "mimic()" of a Dragon command.
4. Finally, switch back to the state stored at step 1.

[lettersandnumbersgithub]

In short:

Store the "state" of the current exclusiveness.
Disable exclusiveness of all grammar. It makes Dragon commands available.
Do the "mimic()" of a Dragon command.
Finally, switch back to the state stored at step 1.

I was thinking the same thing :) But was wondering how quickly exclusivity can be switched on and off?

Each time a rule's exclusiveness is changed, must it be unloaded and reloaded? Or disabled and re-enabled?

If listening for ShellMessages (HSHELL_WINDOWACTIVATED, HSHELL_RUDEAPPACTIVATED, HSHELL_WINDOWCREATED especially) is not possible with Caster, one possible way to switch context when the window changes would be to check the window handle on_end(), so that any time the window changed as a result of a voice command, exclusivity could be changed post voice command and be less impactful, and would only need to be changed on_begin() the times it was changed by keyboard or mouse or an application popped up/stole focus.

Or perhaps Caster could have its own accompanying Python script which runs on its own thread and is entirely separate, much like an AutoHotkey script, which would not be bound by threading issues and could listen for ShellMessages and notify caster to change context/exclusiveness and perform numerous other functions (pun intended :p) asynchronously, perhaps it would help with GUIs also :)

[davidlehub]

I was thinking the same thing :) But was wondering how quickly exclusivity can be switched on and off?

With the new caster system, it seems pretty fast. But event with the old one, when it has a noticeable delay (bcz of the waiting of the grammars to become exclusive), It doesn't that bad. But anyway, i think, it's preferable to wait a little bit (worst case 2 to 3 sec) and have Caster respond more accurately, then been frustrated by no needed vocabularies interfering and executing wrong commands.

[davidlehub]

Each time a rule's exclusiveness is changed, must it be unloaded and reloaded? Or disabled and re-enabled?

If i am not wrong, it disabled and re-enabled, except for the grammar/s of the CCR...

Pretty sure is the same with the new Caster system.

[davidlehub]

Or perhaps Caster could have its own accompanying Python script which runs on its own thread and is entirely separate, much like an AutoHotkey script, which would not be bound by threading issues and could listen for ShellMessages and notify caster to change context/exclusiveness and perform numerous other functions (pun intended :p) asynchronously, perhaps it would help with GUIs also :)

Oh yea, agreed :)

[davidlehub]

"Go To Sleep" "Wake Up" (if also excluded) "Click " "Press "

They are all excluded (all Dragon commands).

(Yeaa, our Dragon is not happy and turns a deaf ear, bcz we excluded him :-D )

As Solution, what i did:

• hmm, i think is more easy to explain with a screen view... i will post a video.

In short:

1. Store the "state" of the current exclusiveness.
2. Disable exclusiveness of all grammar. It makes Dragon commands available.
3. Do the "mimic()" of a Dragon command.
4. Finally, switch back to the state stored at step 1.

So, to use a Dragon command, for example "Go To Sleep", here is a "summary" of what i did (in the old Caster system): https://drive.google.com/open?id=160GLTsiEu3XDfVr-iCtDVgTecqAAvWkf

(you can open that drawing by clikcing on: So u can copy the texts in it, if needed.)

[LexiconCode]

• That "_process_begin" function is called ONLY when the microphone start to dectecting some things (e.g., the user start to speak in to the microphone).

I think it should be fine though because that's how dragonfly monitors window changes for all grammars anyway and I haven't seen really any performance issues with it.

Also according to Dane Grammars shouldn't need to be unloaded before set_exclusiveness() calls take effect, although they do need to be loaded for that method to work. I believe that means we just need to track which grammars we make exclusive.

Last Dane is thinking to include in rules the ability to define exclusiveness. That will work well for grammars of we always want to be exclusive when enabled.

[drmfinlay]

@LexiconCode Thanks for passing on my messages :+1:

It would be pretty useful to have an exclusivity manager for enabling/disabling Dragon's built in rules like this!

It would be possible to implement a Rule.set_exclusiveness() method for setting Dragonfly rules as exclusive/non-exclusive. It would be quite tricky to implement though and couldn't really work well with non-exported rules, which is how CCR usually works.

[lettersandnumbersgithub]

"Go To Sleep" "Wake Up" (if also excluded) "Click [button]" "Press [modifier] [key]"

"Go To Sleep" is also blocked by my free dictation interception "<text>": , so I was able to devise and test this now, despite not yet having the ability to enable exclusiveness. The following are working replacements:

import natlink

"Go To Sleep":         Function(lambda: natlink.setMicState("sleeping") ),
"Mic Off":         Function(lambda: natlink.setMicState("off") ),

As we cannot get the dictation while the microphone is asleep (inconvenient, but good for security and privacy), if "Wake Up": is also blocked, perhaps the only solution will be to disable all exclusiveness when going to sleep and re-enable it again upon waking (easily done with changeCallback())

[LexiconCode]

It would be quite tricky to implement though and couldn't really work well with non-exported rules, which is how CCR usually works.

The way the current implementation works around this is disabling all rules then re-enabling exclusive only rules. There's not much added benefit to this outside of blocking Dragon.

I think the first step implementation here would be to enable exclusivity for all Caster grammars.

Going beyond that to work with CCR is going to be tricky. Honestly beyond my ability to pull off but I could imagine it something like this.

1. Disabling the merged grammars and backing up current enabled grammars.
2. Run any CCR exclusive grammars through merger with a unique exclusive prefix(so it doesn't conflict).
3. Once done with the exclusive mode/grammars, delete their unique merges and restore by enabling the previous date mentioned in step one.

If the system worked as described there wouldn't be the penalty of reemerging everything in the first step when exiting exclusivity it would be just re-enabled

[davidlehub]

If the system worked as described there wouldn't be the penalty of reemerging everything in the first step when exiting exclusivity it would be just re-enabled.

:thumbsup: No reemerging = No delay, especially when the callback "_process_begin" is use to detect "foreground window" changes.

[LexiconCode]

@davidlehub For making exclusivity for all loaded Caster grammars. Simply placing grammar.set_exclusiveness(1) right after grammar.load() as a proof of concept.

To be placed in the following lines MappingRules and MergeRule in GrammarManager.

Now that we know where to place the event we can implement a hook to do more advanced functionality.

I'm a bit short on time @lettersandnumbersgithub could fill you in on some of the other issues like Dragon sleeping and so forth.

[lettersandnumbersgithub]

For making exclusivity for all loaded Caster grammars. Simply placing grammar.set_exclusiveness(1) right before grammar.load() as a proof of concept.

For making exclusivity for all loaded Caster grammars. Simply placing grammar.set_exclusiveness(1) right after grammar.load() as a proof of concept.

:)

[lettersandnumbersgithub]

With this change alone, grammars remain exclusive and active even after putting Dragon to sleep (for which the workaround I described above is required) However I have written the following which appears to solve this problem:

"Go To Sleep":         Function(lambda: GoToSleep() ),

def GoToSleep():
   for grammar in get_engine().grammars:
      grammar.set_exclusiveness(0)
   natlink.setMicState("sleeping")

[LexiconCode]

@davidlehub @lettersandnumbersgithub All right I have a working hook an event for exclusivity for all Caster loaded grammars. Note this does not include Dragonfly grammars as they are not loaded through Casters Grammar Manager.

Take a look here

If you try out the code above you'll need to start dns once. Then go to hooks.toml and set ExclusiveHook = true or say enable exclusive hook then restart DNS again.

I'm curious to know the behavior when enabling the hook after starting caster compared to it's enabled on boot.

I'll take some feedback and questions here before making a pull request.

[davidlehub]

With this change alone, grammars remain exclusive and active even after putting Dragon to sleep (for which the workaround I described above is required) However I have written the following which appears to solve this problem:

"Go To Sleep":         Function(lambda: GoToSleep() ),

def GoToSleep():
   for grammar in get_engine().grammars:
      grammar.set_exclusiveness(0)
   natlink.setMicState("sleeping")

Just some think relate to it, on my mind now: in the function "GoToSleep", maybe storing the "state" of the current exclusiveness (for example, put in a list, all grammars, or rules, been exclusive). So when Dragon is "wake up", we restore the saved state.

Code would be some thing like:

def GoToSleep():

   --> Store/save Things Currently Been Exclusive
   for grammar in get_engine().grammars:
     ....

and then when, for example:

def WakeUp():
    --> reStore Things that was been Exclusive
    ....

:)

[davidlehub]

@LexiconCode

If you try out the code above you'll need to start dns once. Then go to hooks.toml and set ExclusiveHook = true or say enable exclusive hook then restart DNS again.

how to try it? i click on "Create pull request"?

[LexiconCode]

how to try it? i click on "Create pull request"?

Pull request are meant to request merging changes into a repository. In this context and the upstream Caster repository. Great for when you have a new feature you would like to include in Caster.

There's a few ways to go about doing this. Git - you'll need to add my repository has a remote. Once that's done you can check out the branches. See Git Extensions Docs

Zip - Download my branch through the zip file by going to my exclusive branch in my caster fork.

[davidlehub]

have error: AttributeError: 'NoneType' object has no attribute 'file'

Caster: castervoice is up-to-date
Error loading _caster from C:\Users\HP\Documents\Caster\_caster.py
Traceback (most recent call last):
  File "C:\NatLink\NatLink\MacroSystem\core\natlinkmain.py", line 331, in loadFile
    imp.load_module(modName,fndFile,fndName,fndDesc)
  File "C:\Users\HP\Documents\Caster\_caster.py", line 65, in <module>
    control.init_nexus(_content_loader)
  File "C:\Users\HP\Documents\Caster\castervoice\lib\control.py", line 7, in init_nexus
    _NEXUS = Nexus(content_loader)
  File "C:\Users\HP\Documents\Caster\castervoice\lib\ctrl\nexus.py", line 83, in __init__
    self._load_and_register_all_content(rules_config, hooks_runner, transformers_runner)
  File "C:\Users\HP\Documents\Caster\castervoice\lib\ctrl\nexus.py", line 92, in _load_and_register_all_content
    content = self._content_loader.load_everything(rules_config)
  File "C:\Users\HP\Documents\Caster\castervoice\lib\ctrl\mgr\loading\load\content_loader.py", line 57, in load_everything
    rules = self._process_requests(rule_requests)
  File "C:\Users\HP\Documents\Caster\castervoice\lib\ctrl\mgr\loading\load\content_loader.py", line 106, in _process_requests
    content_item = self.idem_import_module(request.module_name, request.content_type)
  File "C:\Users\HP\Documents\Caster\castervoice\lib\ctrl\mgr\loading\load\content_loader.py", line 92, in idem_import_module
    return fn()
  File "C:\Users\HP\Documents\Caster\castervoice\rules\core\utility_rules\window_mgmt_rule.py", line 20, in get_rule
  File "C:\Users\HP\Documents\Caster\castervoice\lib\ctrl\mgr\rule_details.py", line 31, in __init__
    self._filepath = RuleDetails._calculate_filepath_from_frame(stack, 1)
  File "C:\Users\HP\Documents\Caster\castervoice\lib\ctrl\mgr\rule_details.py", line 37, in _calculate_filepath_from_frame
    filepath = module.__file__.replace("\\", "/")
AttributeError: 'NoneType' object has no attribute '__file__'
-- skip unchanged wrong grammar file: C:\Users\HP\Documents\Caster\_caster.p

(I used Git to have files localy.)

[LexiconCode]

have error: AttributeError: 'NoneType' object has no attribute 'file'

delete window_mgmt_rule.pyc

[davidlehub]

I'm curious to know the behavior when enabling the hook after starting caster compared to it's enabled on boot.

Test result:

[+] Mannulaly make "ExclusiveHook = true", in the hooks.toml, exclusiveness works as expected.

[-] When speech "disable exclusive hook", the spec is recognised, but nothing change: exclusiveness is still actived/enabled. In hooks.toml, "ExclusiveHook " still = true. Even after restart Dragon.

[-] After manually change "ExclusiveHook = false", in the hooks.toml , then run Dragon, then speech "enable exclusive hook", the spec is recognised, but nothing change: exclusiveness is still deactived/disabled. "ExclusiveHook " still = false. Even after restart Dragon.

[LexiconCode]

Okay that's pretty much what I expected. There's definitely more work to be done but it's a start. Some enhancements need to be made to the event and hook system for this to become robust.

• Hook could use multiple Events #746
• Hook should have functions that trigger when enabled and disabling the hook. #749

[davidlehub]

Hi guys :) I wonder if there is already a function (or place in the code) that detect foreground/active window changed?

[davidlehub]

what i mean is a LOGIC that detect it. Not the methodes, like get_active_window_title , in castervoice\lib\utilities.py