libvarnam icon indicating copy to clipboard operation
libvarnam copied to clipboard

Published 20 hours ago • verified publisher icon varnamproject

Rust port?

Open asdofindia opened this issue 4 years ago • 6 comments

Is a Rust port of varnam a bad idea? If someone's porting varnam into a different language, what should they do differently?

asdofindia avatar Feb 09 '21 15:02 asdofindia

We(Subin, Kailash and me) had a plan to port Varnam to Go. Also had a conversation with Navaneeth on the architecture as well. AFAIK, core of this project is a lexer(I guess) written in Ruby. It leverages advantages of that language. porting the rule engine will be the a bottleneck I guess. It will be great to port to modern languages, either go or rust.

joicemjoseph avatar Feb 10 '21 03:02 joicemjoseph

If we're going to do a rewrite, some improvements I suggest :

  • No need of a pattern => word DB (learnings file). Instead, just need a word dictionary.
  • Add more patterns to VST (Varnam Symbol Table). Prioritized letters n => ന, ണ. Capitalized N will always give ണ. So pani will give suggestions in priority : പനി, പണി. Currently if only the learnings DB has pani assigned to both words will give the different outputs.
  • When an input say pani is given to varnam, it should tokenize to പനി and പണി using just VST, and then look up the word dictionary to find words starting with പനി and പണി and give additional suggestions.
  • For english words like "Cricket", the tokenization will give bad results, in such cases we can maybe use a pattern => word DB like the current learnings DB.

subins2000 avatar Feb 10 '21 05:02 subins2000

Also, can dynamic libraries be made in Go & Rust ? Can it then be compiled in Android to be used in Indic Keyboard ? A straight replacement ? Is there a JNI equivalent for Go/Rust libraries in Java ? If not, things will be more difficult.

subins2000 avatar Feb 10 '21 05:02 subins2000

There's https://github.com/mozilla/rust-android-gradle for rust in android

asdofindia avatar Feb 10 '21 05:02 asdofindia

Varnam desktop app, varnamd API server is all written in Go. If we're going to do a rewrite, golang will be the better choice for compatibility reasons and maintainability. I saw that there is JNI compatibility with go too.

I hope it will be a straight, simple replacement or others that depend on libvarnam breaks! (Desktop, Indic Keyboard, API) :sweat_smile:

subins2000 avatar Feb 10 '21 15:02 subins2000

@joicemjoseph , you're right about the lexer. https://asd.learnlearn.in/varnam-1/

asdofindia avatar Feb 15 '21 10:02 asdofindia