rust-tinysegmenter
rust-tinysegmenter copied to clipboard
woxtu
Compact Japanese tokenizer
rust-tinysegmenter
Rust implementation of TinySegmenter, which is a compact Japanese tokenizer.
Install
Adding the following to the Cargo.toml in your project:
[dependencies]
tinysegmenter = "0.1"
and import using extern crate:
extern crate tinysegmenter;
Usage
let tokens = tinysegmenter::tokenize("私の名前は中野です");
println!("{}", &tokens.join("|")); // 私|の|名前|は|中野|です
License
Copyright (c) 2015 woxtu
Licensed under the MIT license.
woxtu