React Native

[johnbowdenatfacet]

Just wondering if anyone has integrated this into React Native (ios and android)?

[nshmyrev]

We can take a look on this, what platform is your priority ios or android?

[johnbowdenatfacet]

We can take a look on this, what platform is your priority ios or android?

We're hoping to support both, so both equal priority.

[erksch]

Maybe I could help with this, as I have used this library for a while and already integrated the android part over the react native bridge into our react native app (same for windows). But I have no experience for iOS.

[johnbowdenatfacet]

@erksch that would be great, if we can get both ios and android done and hopefully release as a module would be good. @nshmyrev we might need a hand exposing to IOS.

[LirryPinter]

@erksch Is your usage of the react native brigde for this library somewhere on github? I'm trying to do the same but i'm fairly new to react native and vosk.

[erksch]

@LirryPinter I could make a Gist for the android native module. For iOS please contact @nshmyrev per mail.

[LirryPinter]

@erksch Thank you a lot! I will await the Gist, and will send the email

[erksch]

@LirryPinter

Here is the Gist for the module: https://gist.github.com/erksch/f3d46b478b3912a00a4dc25726d0260c

Things to note:

• Don't forget to write an appropriate package file (refer to the native module docs for that)
• Enable Kotlin for your project, the module is written in Kotlin
• Include Klaxon (for JSON parsing) and VOSK as a dependency to your android project:

dependencies { 
   implementation 'com.beust:klaxon:5.0.1'
   implementation 'com.alphacep:vosk-android:0.3.12'
}

• The module works by sending events to JS like onPartialResults or onResults. The design for the events is the same as the android speech recognizer, you can change things up if you want.
• Write a proper class on JS side to handle the events and use the results as you please
• The module requires you to store a working model in filesDir/models/kaldi. You can use any other directory you like instead that may be easier accessible that the applications' files directory.
• The module is not perfect and you should adapt its API and use of partial results, results and final results to your needs.

Many of these things could be encapsulated by having a nice NPM package, we should definitely do this at some point.

[Lirry18]

@erksch

Thanks so much for the help. Sadly some of the steps seem a bit above my level of expertise.. Well most importantly the first one, because enabling Kotlin, and including the dependencies is already working. I did some googling on how to write a package file but I get mixed ways of doing it which confuses me.

Could you maybe be a bit more specific about how to write a package file for it? Thanks anyway for the gist!

[erksch]

package <your-package>

import com.facebook.react.ReactPackage
import com.facebook.react.bridge.NativeModule
import com.facebook.react.bridge.ReactApplicationContext
import com.facebook.react.uimanager.ViewManager

class KaldiRNPackage : ReactPackage {
    override fun createNativeModules(reactContext: ReactApplicationContext): List<NativeModule> {
        return listOf(KaldiRNModule(reactContext))
    }

    override fun createViewManagers(reactContext: ReactApplicationContext): List<ViewManager<*, *>> {
        return emptyList()
    }
}

Should be it :) Pretty much the same as https://reactnative.dev/docs/native-modules-android#register-the-module

Then use the package in your MainApplication.java/.kt like you see in the docs also:

protected List<ReactPackage> getPackages() {
  List<ReactPackage> packages = new PackageList(this).getPackages();
  packages.add(new KaldiRNPackage());
  return packages;
} 

Then it should be available in JS as:

import { NativeModules } from 'react-native';
const Kaldi = NativeModules.Kaldi;

[Lirry18]

Hi! Thanks so much, things are going relatively smooth now. I can't seem to find the vosk dependency though..

This is the error. I also tried with version 0.3.15 since that is a later one but also no luck..

Any ideas?

[Lirry18]

Actually, I now notice on the github that vosk-android doesn't exist. Only within the api repo or as android-demo.. Could that be the reason?

[nshmyrev]

I can't seem to find the vosk dependency though

You need to add bintray maven repo in gradle:

https://github.com/alphacep/vosk-android-demo/blob/0d0a3ec212fc37e604c3fde644bf4d4ce045b77a/app/build.gradle#L5

[Lirry18]

Thanks! That worked.

My last problem now is here:

My build seems to fail at the kotlin file, it doesn't recognize the version argument for the boolean. I have to note that I did not add a model to the filesdir, could that be the problem?

@erksch Maybe you have a clue?

[erksch]

Sorry there is an error in the gist. Just remove the version parameter in the createModel call in the initialize method.

Edit: Fixed the error in the Gist.

[Lirry18]

@erksch I did that, couldn't find any other reference to this version parameter so thought it was weird. The only error is still the reactContext parameter passed in the CustomKaldiPackage.kt..

[erksch]

A yeah the variable is named wrong use the one that is the method parameter. (Fixed it in the code snippet now)

[Lirry18]

@erksch The app builds now! Coming closer to the endline. The final problem now is that when I import my native modules in App.tsx, the array is empty. I followed all the instructions, and also tried making a wrapper with a module.exports line, and importing it in App.tsx but this also doesn't work since this is a node notation I think.

Did you have this problem as well? My react-native version is 0.63.3 if that information helps..

[erksch]

Have you registered the package in your MainApplication file?

[Lirry18]

My MainApplication file is in java, heared some pp change that in to kotlin as well, but wasn't sure. The code is below, I added the package like you instructed

package com.testmoduleproject;


import com.testmoduleproject.KaldiRNPackage;
import android.app.Application;
import android.content.Context;
import com.facebook.react.PackageList;
import com.facebook.react.ReactApplication;
import com.facebook.react.ReactInstanceManager;
import com.facebook.react.ReactNativeHost;
import com.facebook.react.ReactPackage;
import com.facebook.soloader.SoLoader;
import java.lang.reflect.InvocationTargetException;
import java.util.List;

public class MainApplication extends Application implements ReactApplication {

  private final ReactNativeHost mReactNativeHost =
      new ReactNativeHost(this) {
        @Override
        public boolean getUseDeveloperSupport() {
          return BuildConfig.DEBUG;
        }

        @Override
        protected List<ReactPackage> getPackages() {
          @SuppressWarnings("UnnecessaryLocalVariable")
          List<ReactPackage> packages = new PackageList(this).getPackages();
          packages.add(new KaldiRNPackage());
          // Packages that cannot be autolinked yet can be added manually here, for example:
          // packages.add(new MyReactNativePackage());
          return packages;
        }

        @Override
        protected String getJSMainModuleName() {
          return "index";
        }
      };

  @Override
  public ReactNativeHost getReactNativeHost() {
    return mReactNativeHost;
  }

  @Override
  public void onCreate() {
    super.onCreate();
    SoLoader.init(this, /* native exopackage */ false);
    initializeFlipper(this, getReactNativeHost().getReactInstanceManager());
  }

  /**
   * Loads Flipper in React Native templates. Call this in the onCreate method with something like
   * initializeFlipper(this, getReactNativeHost().getReactInstanceManager());
   *
   * @param context
   * @param reactInstanceManager
   */
  private static void initializeFlipper(
      Context context, ReactInstanceManager reactInstanceManager) {
    if (BuildConfig.DEBUG) {
      try {
        /*
         We use reflection here to pick up the class that initializes Flipper,
        since Flipper library is not available in release mode
        */
        Class<?> aClass = Class.forName("com.testmoduleproject.ReactNativeFlipper");
        aClass
            .getMethod("initializeFlipper", Context.class, ReactInstanceManager.class)
            .invoke(null, context, reactInstanceManager);
      } catch (ClassNotFoundException e) {
        e.printStackTrace();
      } catch (NoSuchMethodException e) {
        e.printStackTrace();
      } catch (IllegalAccessException e) {
        e.printStackTrace();
      } catch (InvocationTargetException e) {
        e.printStackTrace();
      }
    }
  }
}

[erksch]

Side note: Java is fine and will not make any problems. But the Android world is moving towards Kotlin so make sure to get it in your vocabulary.

Seems good. The NativeModules should be an object that is never empty it contains dozens of default modules. Can you check that you imported and printed it correctly?

[Lirry18]

Right now I import it like this:

import { NativeModules } from 'react-native' console.log(NativeModules)

Did some googling and found that more people have issues: https://github.com/facebook/react-native/issues/26813

Going to look at it, maybe its a react issue

[Lirry18]

@erksch Seems a common problem with react native v > 0.60, can I ask you which version you are using? Maybe I can try downgrading it to that version and see if the problem persists.

[erksch]

I use 0.63. I wouldn't downgrade below 0.60 because there are many differences between 0.5 and 0.6 which may break your project. Are you using expo by any chance? Where do you view your logs, do you use the Chrome debugger?

[Lirry18]

I also used that hmm. No its a react-native cli project so no expo. Just tried the debugger, and on chrome it shows an [object Object] instead of an empty array (in the 'normal' node logger). So thats a bit better at least. Still lost on how to solve this problem

[Lirry18]

Finally found it I think! I can log it in the debugger as "Kaldi", not KaldiVOSKModule! Hopefully it will work outside of debugging in Chrome. Will keep you posted!

Edit: Also works outside the debugging environment now. Seems that a lot of developers have this problem, its probably somewhere in the core of react native.

[erksch]

Ah yeah, you can see in the Gist the module's name is "Kaldi" not "KaldiVOSKModule". https://gist.github.com/erksch/f3d46b478b3912a00a4dc25726d0260c#file-kaldivoskmodule-kt-L40

[Lirry18]

Haha all that hussle for something so small #programming. I now get the error model creation failed. The files are in the highest level (so on the level of node_modules, android, ios etc) in a map called models/kaldi. The files are maps with "am", "conf", "graph" and "ivector".

Must I place them next to the kotlin files or in main?

[erksch]

No the filesDir is not the highest level. Sorry, but this get's rather complicated to explain. Inform yourself about storing files in Android: https://developer.android.com/training/data-storage.

About filesDir here: https://developer.android.com/training/data-storage/app-specific Choose a location that you understand the best and put the files there and implement the path into the Kaldi module.

The filesDir setup I use may be not the one you want, try to use assets or something.

Maybe some general advice. I have nothing against having ambitious projects that are beyond your knowledge so you can grow on them, but take help from others only if you are definitely stuck, not when the next error occurs. If you spent 3 hours understanding the context around an error in order to solve it in the end, then you'll actually gain knowledge. I hope that didn't sound too preachy :D

[Lirry18]

I see! Im confused about why I am not getting the error '..path doesn't exist'. It doesn't matter what I put in, or what I log the promise always is rejected somehow.