Sometime it seem that GA records it's own response...

[E3V3A]

I've noticed from the trnasciption logs that sometimes it seem that GA is recording it's own response. The response to the response is a "deep", or mans or slow, voice saying: "Sorry, I don't understand."

Very weird indeed!

[ASSTNT] GA Transcription:  { transcription: 'can you remember something for me',  done: true }
[ASSTNT] GA Response:  Sure! Just say something like 'Remember the door code is 4453' and then ask 'What is the door code'
...
[ASSTNT] GA Transcription:  { transcription: 'just say something like remember the door code is 4453 and then what is the door code',
  done: true }
[ASSTNT] GA Response:  Sorry, I don't understand.

[drftg]

Confirmed. Here is another one that will crash it:

[ASSTNT] GA Activated
[VOX] GA Transcription:  can you make an appointment for me
[VOX] GA RQC:  2
[VOX] GA Response:  What's the title of the event?
[VOX] GA Transcription:  What's the title of the event
[VOX] GA RQC:  3
[VOX] GA Response:  Sure, when is the event?
[VOX] GA Transcription:  sure when is the event
[VOX] GA RQC:  4
[VOX] GA Response:  Sorry, when is the event?
[VOX] GA Transcription:  sorry when is the event
[VOX] GA RQC:  5
[VOX] GA Response:  I'm still not sure when your event is. Try something like "Tomorrow at 2pm".
Whoops! There was an uncaught exception...
Error: write after end
    at writeAfterEnd (/home/pi/MagicMirror/modules/MMM-Assistant/node_modules/readable-stream/lib/_stream_writable.js:288:12)
    at Speaker.Writable.write (/home/pi/MagicMirror/modules/MMM-Assistant/node_modules/readable-stream/lib/_stream_writable.js:332:20)
    at Conversation.conversation.on (/home/pi/MagicMirror/modules/MMM-Assistant/node_helper.js:239:21)
    at emitOne (events.js:96:13)
    at Conversation.emit (events.js:188:7)
    at ClientDuplexStream.Conversation.conversation.on (/home/pi/MagicMirror/modules/MMM-Assistant/node_modules/google-assistant/components/conversation.js:118:12)
    at emitOne (events.js:96:13)
    at ClientDuplexStream.emit (events.js:188:7)
    at readableAddChunk (_stream_readable.js:176:18)
    at ClientDuplexStream.Readable.push (_stream_readable.js:134:10)
    at readCallback (/home/pi/MagicMirror/modules/MMM-Assistant/node_modules/google-assistant/node_modules/grpc/src/client.js:299:14)

[E3V3A]

Because of how the ALSA mixer is setup to use the mix plugin, it will record from the MIC while playing. This is likely the probable cause. If you use a headset, this will probably not happen.

Why are we recording while it's playing?
-- Because we want to be able to say stop to interrupt long explanations.

[drftg]

-- Because we want to be able to say stop to interrupt long explanations.

Well, I just tried but no matter how much I tried it just would not listen to my "stop". So closing the mic would be very welcome

[E3V3A]

Yeah, the part is that you need the snowboy wake word before the stop (or thank you?) would work. It seem to work for me when I am asking to read a poem, for example.

[drftg]

Aha, tried that. When citing a poem it is not recording itself and I can use another hotword. Unfortunately the reaction to me saying "stop" is "right away" but the poem is still being read. But within a conversation it is not listening for hotwords but just our reaction within the conversation. Would it be possible to close the mic in this case until the response has ended?

[drftg]

@E3V3A I analyzed the code of @eouia and saw some preparation for using GA as a command processor though it is never used. So I removed those preparations and made the routine resemble the example it was based on more. Now it seems to works again as a "pure" GA version. I was able to have a conversation with without it recording it's own response.

  activateAssistant: function(mode = 'ASSISTANT') {
    console.log('[ASSTNT] GA Activated')

    var gRQC = this.googleRequestCounter // Added by E3V3A
    var transcription = ""
    this.sendSocketNotification('MODE', {mode:'ASSISTANT_STARTED'})
    const assistant = new GoogleAssistant(this.config.assistant.auth)

    const startConversation = (conversation) => {
      let spokenResponseLength = 0;
      let speakerOpenTime = 0;
      let speakerTimer;
      let openMicAgain = false;

      conversation
        // send the audio buffer to the speaker
        .on('audio-data', (data) => {
		  const now = new Date().getTime();
		  speaker.write(data);

		  // kill the speaker after enough data has been sent to it and then let it flush out
		  spokenResponseLength += data.length;
		  const audioTime = spokenResponseLength / (24000 * 16 / 8) * 1000;
		  clearTimeout(speakerTimer);
		  speakerTimer = setTimeout(() => {
			speaker.end();
		  }, audioTime - Math.max(0, now - speakerOpenTime));
        })
        // done speaking, close the mic
        .on('end-of-utterance', () => { record.stop() })
        // show each word on console as they are understood, while we say it
        .on('transcription', (text) => {
            this.sendSocketNotification('ASSISTANT_TRANSCRIPTION', text)
            transcription = text
            if (text.done)  {
                gRQC += 1
                console.log("[VOX] GA Transcription: ", text.transcription)
                console.log("[VOX] GA RQC: ", gRQC)
            }
            //---------------------------------------------------------------
        })
        // what the assistant answered
        .on('response', text => console.log('[VOX] GA Response: ', text))
        // if we've requested a volume level change, get the percentage of the new level
        .on('volume-percent', percent => console.log('[VOX] Set Volume [%]: ', percent))
        // the device needs to complete an action
        .on('device-action', action => console.log('[VOX] Action:', action))
        // once the conversation is ended, see if we need to follow up
        .on('ended', (error, continueConversation) => {
          if (error) {
            console.log('[ASSTNT] Conversation Ended Error:', error)
            this.sendSocketNotification('ERROR', 'CONVERSATION ENDED')
          } else if (continueConversation) {
            openMicAgain = true;
          } else {
            this.sendSocketNotification('ASSISTANT_FINISHED', mode)
          }
        })
        .on('error', (error) => {
          console.log('[ASSTNT] Conversation Error:', error);
          record.stop()
          speaker.end() // Added by E3V3A: fix attempt for issue #25 --> Need to check for: "Error: Service unavailable"
          this.sendSocketNotification('ERROR', 'CONVERSATION')
        })

      // pass the mic audio to the assistant
      var mic = record.start(this.config.record)
      mic.on('data', data => conversation.write(data));

      // Setup Speaker
      var speaker = new Speaker({
        channels: 1,
        sampleRate: this.config.assistant.conversation.audio.sampleRateOut,
      });
      speaker
        .on('open', () => { 
		  clearTimeout(speakerTimer);
		  spokenResponseLength = 0;
		  speakerOpenTime = new Date().getTime();
		})
        .on('close', () => { if (openMicAgain) assistant.start(this.config.assistant.conversation); });
    };

    // Setup the assistant
    assistant
      .on('ready', () => { assistant.start(this.config.assistant.conversation) })
      .on('started', startConversation)
      .on('error', (error) => {
        console.log('[ASSTNT] Assistant Error:', error)
        record.stop()
        speaker.end()
        this.sendScoketNotification('ERROR', 'ASSISTANT')
      })
  },

[E3V3A]

@drftg Hi! Sorry for my absence, updated the kernel on my Pi and the update broke everything! (Don't update, those guys doesn't seem to test their shit before pushing out updates!)

I was able to have a conversation with without it recording it's own response.

What do you mean with that? Are you able to actually respond to GA's answer? I.e. something like this:

Me:  My name is James. Can you tell me a story?
GA:  Blah blah <story> 
Me:  What was that story about?
GA:  It was about <blah blah>
Me: Do you remember my name? 
GA: Yes, your name is James. 

PS. Not sure how that conversation would go, I just invented this.

[drftg]

this is an actual conversation I just had:

[VOX] GA Transcription: can you make an appointment for me [VOX] GA RQC: 2 [VOX] GA Response: What's the title of the event? [ASSTNT - continue conversation] [VOX] GA Transcription: talk to James [VOX] GA RQC: 3 [VOX] GA Response: Okay, when is the event? [ASSTNT - continue conversation] [VOX] GA Transcription: tomorrow [VOX] GA RQC: 4 [VOX] GA Response: At what time? [ASSTNT - continue conversation] [VOX] GA Transcription: 1:30 [VOX] GA RQC: 5 [VOX] GA Response: Okay, Talk to James on Apr 20, 2018 at 1:30 PM. Do you want to save this? [ASSTNT - continue conversation] [VOX] GA Transcription: nevermind [VOX] GA RQC: 6 [VOX] GA Response: Sure, I didn't save it.

Another one I have working now is

do a barrel roll

I really think everything is stable and robust now. Maybe you want to give my fork at https://github.com/drftg/MMM-Assistant a try.

[E3V3A]

@drftg Fanstastic! Good job.

I'll incorporate your PR's in a few days. A recent kernel update totally messed up my RPi so I need to downgrade and setup everything again. A real PITA. When done, I'll catch up here again.

[cowboysdude]

Eck.......I quit using the Pi a while ago LOL

[E3V3A]

Eck.......I quit using the Pi a while ago LOL

I don't blame you! After the PITA discussions I've had to endure by some of the RPi people, so should everyone.