Tests for invalid phonemes fallbacks fail on latest pocketsphinx

[PureTryOut]

I'm running the tests on Alpine Linux using system packages and Python 3.8.

I currently have all tests succeeding except for PocketSphinxTest.testInvalid and LocalRecognizerInitTest.testListenerConfig. It seems the fallback for invalid phonemes isn't working.

============================================================= FAILURES ==============================================================
___________________________________________________ PocketSphinxTest.testInvalid ____________________________________________________

self = <test.unittests.client.test_hotword_factory.PocketSphinxTest testMethod=testInvalid>

    def testInvalid(self):
        config = {
            'hey Zeds': {
                'module': 'pocketsphinx',
                'phonemes': 'ZZZZZZZZZ',
                'threshold': 1e-90
            }
        }
        p = HotWordFactory.create_hotword('hey Zeds', config)
>       self.assertEqual(p.phonemes, 'HH EY . M AY K R AO F T')
E       AssertionError: 'ZZZZZZZZZ' != 'HH EY . M AY K R AO F T'
E       - ZZZZZZZZZ
E       + HH EY . M AY K R AO F T

test/unittests/client/test_hotword_factory.py:43: AssertionError
------------------------------------------------------- Captured stdout call --------------------------------------------------------
2020-05-06 14:10:53.204 | INFO     |  7488 | mycroft.client.speech.hotword_factory:load_module:386 | Loading "hey Zeds" wake word via pocketsphinx
____________________________________________ LocalRecognizerInitTest.testListenerConfig _____________________________________________

self = <test.unittests.client.test_local_recognizer.LocalRecognizerInitTest testMethod=testListenerConfig>
mock_config_get = <MagicMock name='get' id='139759354767968'>

    @patch.object(Configuration, 'get')
    def testListenerConfig(self, mock_config_get):
        """Ensure that the fallback method collecting phonemes etc.
        from the listener config works.
        """
        test_config = base_config()
        mock_config_get.return_value = test_config
    
        # Test "Hey Mycroft"
        rl = RecognizerLoop()
        self.assertEqual(rl.wakeword_recognizer.key_phrase, "hey mycroft")
    
        # Test "Hey Victoria"
        test_config['listener']['wake_word'] = 'hey victoria'
        test_config['listener']['phonemes'] = 'HH EY . V IH K T AO R IY AH'
        test_config['listener']['threshold'] = 1e-90
        rl = RecognizerLoop()
        self.assertEqual(rl.wakeword_recognizer.key_phrase, "hey victoria")
    
        # Test Invalid"
        test_config['listener']['wake_word'] = 'hey victoria'
        test_config['listener']['phonemes'] = 'ZZZZZZZZZZZZ'
        rl = RecognizerLoop()
>       self.assertEqual(rl.wakeword_recognizer.key_phrase, "hey mycroft")
E       AssertionError: 'hey victoria' != 'hey mycroft'
E       - hey victoria
E       + hey mycroft

test/unittests/client/test_local_recognizer.py:76: AssertionError

[forslund]

Python 3.8 passes all tests on Travis, gonna test it locally as well to confirm.

Do you know which version of pocketsphinx and python-pocketsphinx is being used?

[PureTryOut]

0.1.15, which is probably the problem here, being 15 point releases further :wink:

[forslund]

Did a quick test and upgraded pocketsphinx resulting in me getting the exact same issue so definitely the issue.

[PureTryOut]

Cool, at least I now know where to look. I'll investigate!

To be honest I'm amazed this is the only problem that comes with the newer versions, seeing how strict Mycroft normally keeps their dependencies

[forslund]

Lol yeah, me too! We've had our fair share of breakage though when doing updates (py2 to 3 was rough, and upgrading from the arcane tornado version to the newer one was quite annoying)

[PureTryOut]

Interesting enough release 0.1.5 through 0.1.15 was done in a space of 3 days, and the tests still break on 0.1.3 which was the first release available on PyPi after 1.0.

0.1.0 is actually from June 2016, so I guess the dep never has been upgraded in mycroft-core since the public release? At least now I know the breakage is somewhere between the 4th of June 2016 and the 12th of September 2016 (between 0.1.0 and 0.1.3).

[PureTryOut]

Hmpf, the problem isn't in pocketsphinx-python itself, it's in the submodule pocketsphinx it uses. If I use pocketsphinx-python 0.1.0 but with newer submodules, the tests break in the same way. But if I use it with the submodules that come with that release, it works fine. I guess that is the issue of using libraries without releases...

In fact, the breakage is between https://github.com/cmusphinx/pocketsphinx/commits/ec89abd9d2e976019338ecae7d6cafeecdb95eb2 and https://github.com/cmusphinx/pocketsphinx/commits/a60982363101704eca342e7e0920754090cd49b1

pocketsphinx-python 0.1.15 with the submodules from 0.1.0 actually makes these tests succeed!

[PureTryOut]

Gotcha, https://github.com/cmusphinx/pocketsphinx/commit/08e367ef349983095ed5fd73f874329069628ac6 is the commit that broke it. I'm not sure how this breaks things, but maybe it's obvious to someone?

[PureTryOut]

Ah, seems https://github.com/MycroftAI/mycroft-core/blob/dev/mycroft/client/speech/hotword_factory.py#L108 is outdated and should now be config.set_string('-kws', <path to some kws file>).

https://github.com/cmusphinx/pocketsphinx/commit/08e367ef349983095ed5fd73f874329069628ac6#diff-0376830e1b8c31e50aa1f84206085a41

[forslund]

Nice find, so instead of passing a keyphrase we need to write the keyphrase and threshold to a tmp file and pass that in? Or is it somehow similar to the -dict file that's used to pass in phonemes and word mappings (I think it is atleast)?

[PureTryOut]

Well the test file used by pocketsphinx contains just a few things:

anything /1e-10/
something
bad line / here
just bad line /
forward

non_existign_word

So it seems like it can be a tmp file, but I'm not sure, I don't know pocketsphinx :joy:

[forslund]

I did a test changing the -keyphrase to -kws and a similar file but gets the same issue. I wonder if there's a way to explicitly verify the phonemes validity other than expecting an error from the Decoder creation. But I may also just doing something wrong

[forslund]

Hmm https://cmusphinx.github.io/wiki/faq/ suggests that -keyphrase still is valid...

[PureTryOut]

Hmm maybe. I guess -keyphrase is always a single phrase, where -kws may contain multiple phrases.

Then it probably goes wrong in verifying the keyphrase it's passed, since that logic has changed.

[forslund]

If you got other things to dig into I can try to spend some time finding some solution to this tomorrow or so.

[PureTryOut]

I have tons of things to do haha. Main problem here is that this is a bit out of my league, so I would love it if you could take a look at it.

[PureTryOut]

@forslund did you have any luck with this by any chance?

[forslund]

I haven't gotten all the way through it. I may have a workaround half-done (if it works)

[PureTryOut]

@forslund how is your workaround going?

[forslund]

Well my first attemt didn't work at all. Sort of forgot about it recently, sorry. Gonna give it some extra attention...