PocketSphinx + Google SpeechRecognition + Android = NO

[Villmer]

I'm now trying to create an android app that will listen for a trigger word, such as "VOICE" and then initiate Google SpeechRecognition. I've built the entire application and tested it in a DESKTOP browser and it works perfectly. (The project is HTML5)

However, when I move this project to Android there is a conflict. I CAN successfully recognize the trigger word but when I then try to initiate web speech recognition it won't work. It appears that the audio (mic) is still locked to pocketSphinx.js. I have tried stopping the recognizer before initiating web speech but no solution worked - I tried using stopRecording() and postRecognizerJob({command: 'stop'}) but neither worked. Is there a way to "release" pocketSphinx (and reinitiate it) so other applications can access the microphone?

[Villmer]

I see someone else has had the same issue: http://sourceforge.net/p/cmusphinx/bugs/396/ Has there been any solution to this on Android?

[Villmer]

By the way, the Google Web SpeechRecognition does TRY to begin (you hear it beep) but then it immediately turns off (another beep) as it encounters a conflict accessing the mic.

How can I (temporarily) shot off pocketsphinx.js (recorder) to free up the mic? Can I?

[syl22-00]

I do not really understand what you are doing and your issues do not seem to be related to pocketsphinx.js.

1. Your link points to an issue in pocketsphinx, not pocketsphinx.js.
2. pocketsphinx.js runs in the browser, the platform on which the browser runs does not matter.
3. If you find issues with the recorder, you might want to look at forums around webRTC, getUserMedia, and the browser you are using. Also, you can try with different web browsers.

[Villmer]

1. Yes, I know the link points to a pocketsphinx issue. However, the issue is identical for HTML5 /Javascript based apps running in Android (via Cordova)
2. The pocketSphinx.js + Google combination works fine in the desktop browser. Using the EMBEDDED Chrome browser (Crosswalk) in Android, however, does NOT work.

On Android, the mic can only be utilized by one thing at a time. It's either Google (SpeechRecognition) or pocketSphinx.js. (not at the same time). Because pocketSphinx.js is such a complex platform, I was just curious if there was a way to just turn it OFF (and back ON again) to free up the audio.

None of the following successfully stopped pocketSphinx so I could use Google SpeechRecognition:

1. audioContext.close()
2. postRecognizerJob({command: 'stop'})
3. stopRecording() ---> function stopRecording() { if(recorder && recorder.stop()){}};

[nshmyrev]

If you want to run on Android, pocketsphinx.js will certainly be too slow

You need to create a simple Cordova plugin. It's not very complex, you just integrate pocketsphinx-android jar and jni library and call them with javascript. Recording is better performed on java side as well, not in javascript.

[Villmer]

@nshmyrev Actually the speed of pocketsphinx.js is very fast on my Huawei p8 Max. If performance could improve through a more elegent integragtion that would be great. However, I'm not too familiar with developing cordova plugins and don't have the time to go that deep. I had hoped between pocketsphinx.js and google api I would have found a reasonably solid solution that allowed continual, high-performance voice recognition - but, alas no. There are some cordova plugins that claim "continual speech recognition" but they, like all others, expire after 60 seconds (the maximum Google permits you).

[syl22-00]

If your target is a Cordova app, there is really no point using pocketsphinx.js. Not only would it be slow, but it might suck a lot of memory. You could also consider the Ispikit Cordova plugin. It is not free, but for your app (with limited sentence length), the free version might be enough. https://github.com/ispikit/ispikit-cordova

[Villmer]

Pocketsphinx.js works fine on Android. In fact, it runs quite fast on my Huawei p8 Max.

This issue is getting it to turn on and off (so other voice technology can work as well)

On Fri, Jan 22, 2016 at 6:19 PM, Sylvain Chevalier <[email protected]

wrote:

If your target is a Cordova app, there is really no point using pocketsphinx.js. Not only would it be slow, but it might suck a lot of memory. You could also consider the Ispikit Cordova plugin. It is not free, but for your app (with limited sentence length), the free version might be enough. https://github.com/ispikit/ispikit-cordova

— Reply to this email directly or view it on GitHub https://github.com/syl22-00/pocketsphinx.js/issues/51#issuecomment-173982993 .

[Villmer]

How do I tell pocketphoenix.js to COMPLETELY stop, including closing it's audio context and so forth? I need to free up the mic so Google to access it.

These did not entirely work:

1. audioContext.close()
2. postRecognizerJob({command: 'stop'})
3. stopRecording() ---> function stopRecording() { if(recorder && recorder.stop()){}};

On Fri, Jan 22, 2016 at 6:19 PM, Sylvain Chevalier <[email protected]

wrote:

If your target is a Cordova app, there is really no point using pocketsphinx.js. Not only would it be slow, but it might suck a lot of memory. You could also consider the Ispikit Cordova plugin. It is not free, but for your app (with limited sentence length), the free version might be enough. https://github.com/ispikit/ispikit-cordova

— Reply to this email directly or view it on GitHub https://github.com/syl22-00/pocketsphinx.js/issues/51#issuecomment-173982993 .

[syl22-00]

pocketsphinx.js, the recognizer, itself does not interact with the microphone, it just gets audio samples and processes them.

Again, as I said, "you might want to look at forums around webRTC, getUserMedia, and the browser you are using." since the microphone access is created with

var input = audioContext.createMediaStreamSource(stream);

I'd try to play around with that, such as calling disconnect or just setting it to null. If that does not help, I'd look at the MediaStream used to create the node.

[Villmer]

I'll try setting "input" as a global variable and, just before calling Google SpeechRecognition object, I'll try setting input.disconnect() or input = null;

[Villmer]

Nothing worked.

So, here is what we have for the pocketsphinx.js / audio recorder code...

---

input = audioContext.createMediaStreamSource(stream); var audioRecorderConfig = {errorCallback: function(x) {}}; recorder = new AudioRecorder(input, audioRecorderConfig);
if (recognizer) {recorder.consumers = [recognizer];} recorderReady = true;}, function(e) {}); }

---

We are dealing with the following variables (objects): "audioContext", "input" and "recorder". On the google side, we have the "recognition" object --> recognition = new SpeechRecognition() To start Google Speech Recognition we call "recognition.start();"

So, the goal here is to STOP or PAUSE the pocketsphinx voice and start the Google voice. Here is the code and the things I've tried to do before calling the recognition.start() function......

if(String(n[n.length - 1])=="VOICE"){ $("#mic").show(); // TRY TO CLOSE / STOP AUDIO RECORDER/ P.S. HERE...
//input=null; //input.close() //input.stop() //recorder.stop(); //audioContext.close() //audioContext.suspend().then(function(){}); //audioContext.close().then(function(){}); //audioContext.suspend().then(function(){recognition.start();}); //audioContext.close().then(function(){recognition.start();}); //stopRecording() //postRecognizerJob({command: 'stop'}) recognition.start(); } // START GOOGLE

All of the things I've tried are commented out. Nothing works.

[Villmer]

Side question: is there a way to access confidence values with grammars in pocketsphinx.js?

[miguelmota]

I think you need to call .stop() on the MediaStream object to completely stop streaming from that media channel.

var localStream = null;
navigator.getUserMedia({
  audio: true
}, function(stream) {
  localStream = stream;
  // ...
}, function(error) {});

// later
localStream.stop();
recognition.start();

[obrienj970]

A bit different but I have an Android C# library (built with Xamarin) that incorporates PocketSphinx Java. I got it from a third party

I use it in a service to listen for a keyword and that works fine.

When I get the keyword I cancel and stop the PocketSphinx recognizer and invoke Android's "RecognizerIntent.ActionRecognizeSpeech" to allow Android/Google to get the spoken text after a beep.

I am using PocketSphinx as a work around to the Google VR inability to do continuous recognition even at the API level. Well you can if you write a loop and enjoy a beep every 5 seconds or so.

This all works great with no microphone conflict at all.

However, if try to use RecognizerIntent.ActionVoiceSearchHandsFree or Intent("android.intent.action.VOICE_ASSIST") I get the microphone conflict.

I believe that this is an Android/Google Search Voice problem in that they appear to take a shortcut accessing the microphone which leads to the conflict.

It's not a permissions issue either, if I start my service but don't start PocketSphinx, then the Google Search Voice facilities work. As soon as I use PocketSphinx, Google Search voice support fails.

If I stop my server completely, Google Search Voice still fails. Only a reboot clears it.

As a work around, I am going to use the Google Custom Search support, accept the query in my VR code, send it to via Custom Search, and read back the responses or display them or both.

For my purposes, a 999 queries/month (free) is fine.

BTW, this issue is discussed in many forums with a number of different scenarios most of which don't involve PocketSphinx. In some cases it is device specific. A fix was supposed to be out last month but my Google App is up to date and it still fails. Strangely it used to work, about 2 months ago.

Jim

[syl22-00]

@obrienj970 your issue is not related to pocketsphinx.js. Pocketsphinx repo is here: https://github.com/cmusphinx/pocketsphinx

[dplusic]

The way @miguelmota said works well. stream.stop is deprecated, so please note that link.