Speech Command Tutorial

[khanhlvg]

@farmaker47 thank you for agreeing to help with the speech command tutorial!

Background

• Speech command sample is already available on GitHub but there's no good documentation explaining how to use it.
• We'd like you to help with creating and end-to-end tutorial on how to train the TFLite model and deploy it to a mobile app.

Our ask

• Turn these scripts into a single Colab notebook that demonstrate how to train a speech command TFLite model. You can refer to this tutorial for idea.
• In the end of the notebook, add an instruction on how to integrate the model into the speech_command Android (and iOS, if possible) app.

[farmaker47]

I started working on it! Thanks

[farmaker47]

Hi Khanh,

@khanhlvg There is a broken link inside project... at this file line 20.

Has this tutorial moved to new location?

Thanks

[khanhlvg]

I just learnt that there's an engineer in TFLite team is now working on rebuilding the speech command sample using TF2 and that may overlap with the tutorial that you're working on.

Do you mind pausing the effort for now? Sorry for the back and forth...

There is a broken link inside project... at this file line 20.

I think it was taken down from TF.org because it's a TF 1 tutorial. I think it's the same one with this document on GitHub. https://github.com/tensorflow/docs/blob/master/site/en/r1/tutorials/sequences/audio_recognition.md

[farmaker47]

Yes no problem,

I saw that code was with TF version 1.14 and I was wondering if everything was going to compile. It is a wise decision to pause the effort for creating the colab notebook for now...but nothing will stop me from converting Speech commands app to Kotlin... :) :) :)

Thanks

[khanhlvg]

but nothing will stop me from converting Speech commands app to Kotlin... :) :) :)

That's nice! There won't be overlapping on that so please proceed :) Once the training code is converted to TF2, we can connect it with the new Kotlin app that you are going to build.

[farmaker47]

Hi Khanh,

I want to ask if I can use the same colors, TensorFlow logos and layouts inside Speech commands Kotlin app. Is it a problem?

[farmaker47]

Update. App is almost ready. Kotlin, MVVM, DI, Coroutines.

Working already:

• When user says one of the screen words the specific boxes are highlighted.
• UI is exactly the same

Not working yet:

• Minus and plus button for threads

Also I want to ask if I have to unhide selection for NNAPI delegate and make it work. This option is hidden and apparently never completed. Do you think I have to show it now?

Thanks

[khanhlvg]

I want to ask if I can use the same colors, TensorFlow logos and layouts inside Speech commands Kotlin app. Is it a problem?

Sure, feel free to use the TF colors.

Also I want to ask if I have to unhide selection for NNAPI delegate and make it work. This option is hidden and apparently never completed. Do you think I have to show it now?

If it wasn't shown in the current app UI and not currently used then I think it's fine to keep it that way. Delegate support would be nice to have but I haven't tested if the model is supported by any of the current delegates.

[farmaker47]

I have already finished app. Now minus and plus button select number of threads and this is kept upon rotation. Also application is working during configuration changes.

I have created a landscape layout file:

I will work the READ.ME file for some more info for the users.

Please build to confirm that everything is working and get back with some suggestions of improvement.

Thank you

[farmaker47]

I played with some delegates

It seems that GPU is not supported as there is no improvement of speed.

But I think NnApiDelegate is supported as it give results....lower inference time but it runs with no problems. CPU time 100ms with 1 thread ---------- 60ms with 2 threads ----------- no difference with more threads

NnApiDelegate time 350ms with 1 thread ----------------------- no difference with more

[khanhlvg]

It's very likely that NNAPI CPU implementation is slower, especially if you're using an older phone because NNAPI implementation is included in Android while TFLite CPU implementation can be updated over time.

Given that the delegates don't provide any performance advantage, let's exclude it from the UI then.

[sayakpaul]

@farmaker47 are you still working on this project?

[farmaker47]

No I am not Sayak . I have just finished the android application.

[sayakpaul]

I think we could still include the application as a demo app within our README.