libavrophonetic
libavrophonetic copied to clipboard
Go module providing classic (rule-based) and dictionary backed transliterators for Avro Phonetic
libavrophonetic
[Work in Progress]
Go module providing classic (rule-based) and dictionary backed transliterators for Avro Phonetic.
Optimizations
This implementation is also the fastest dictionary based suggestion generator as far as I know. Primarily because this does not scan through the dictionary looking for regular-expression match and use a Trie instead.
Comparing apples to oranges (because why not), this is ~100 times faster than previous JavaScript and regular-expression based suggestion generator (tested in Node.js env).
Demo
This module is intended to be used as a library.
However, for quickly checking the output there is a demo CLI. Run the following command:
go run ./cmd/avrophoneticdemo shadhinota
Installation
go get -u github.com/mugli/libavrophonetic
Usage/Examples
package main
import (
"fmt"
"os"
"github.com/mugli/libavrophonetic/databasedconv"
"github.com/mugli/libavrophonetic/rulebasedconv"
)
func main() {
input := "bangla"
rulebasedConverter := rulebasedconv.NewConverter()
databasedConverter, _ := databasedconv.NewConverter() // ignoring error for brevity
rulebasedOutput := rulebasedConverter.ConvertWord(input)
databasedOutput := databasedConverter.ConvertWord(input)
fmt.Printf("(Rulebased conversion) %s = %s \n", input, rulebasedOutput) // বাংলা
fmt.Printf("(Databased conversion) %s = %v \n", input, databasedOutput) // [বাংলা বাঙলা]
}
API Documentation
https://pkg.go.dev/github.com/mugli/libavrophonetic
Running Tests
To run tests/see coverage, run the following commands:
make test
make test-cover
Data generation
Instead of using plain text data-files, this module uses a gob encoded files for faster data loading (aka, Trie generation).
Also, the gob files gets embedded with the binary during compile time using embed package introduced in Go 1.16.
If you change source files (having soure-
prefix in the filenames) in ./data
directory, run the following command to re-generate the binary data files:
make generate-data
Authors and Acknowledgement
The git blame is not showing Tahmid's name in this repo because this was started from scratch, but both me and @tahmidsadik had a lot of fun making the initial prototype for this.