SuPar-UniDic

Tokenizer, POS-tagger, lemmatizer, and dependency-parser for modern and contemporary Japanese with BERT models.

Basic usage

>>> import suparunidic
>>> nlp=suparunidic.load()
>>> doc=nlp("太郎は花子が読んでいる本を次郎に渡した")
>>> print(type(doc))
<class 'spacy.tokens.doc.Doc'>
>>> print(suparunidic.to_conllu(doc))
1	太郎	タロウ	PROPN	名詞-固有名詞-人名-名	_	12	nsubj	_	SpaceAfter=No|Translit=タロー
2	は	は	ADP	助詞-係助詞	_	1	case	_	SpaceAfter=No|Translit=ワ
3	花子	ハナコ	PROPN	名詞-固有名詞-人名-名	_	5	nsubj	_	SpaceAfter=No|Translit=ハナコ
4	が	が	ADP	助詞-格助詞	_	3	case	_	SpaceAfter=No|Translit=ガ
5	読ん	読む	VERB	動詞-一般	_	8	acl	_	SpaceAfter=No|Translit=ヨン
6	で	て	SCONJ	助詞-接続助詞	_	5	mark	_	SpaceAfter=No|Translit=デ
7	いる	居る	AUX	動詞-非自立可能	_	5	aux	_	SpaceAfter=No|Translit=イル
8	本	本	NOUN	名詞-普通名詞-一般	_	12	obj	_	SpaceAfter=No|Translit=ホン
9	を	を	ADP	助詞-格助詞	_	8	case	_	SpaceAfter=No|Translit=オ
10	次郎	ジロウ	PROPN	名詞-固有名詞-人名-名	_	12	obl	_	SpaceAfter=No|Translit=ジロー
11	に	に	ADP	助詞-格助詞	_	10	case	_	SpaceAfter=No|Translit=ニ
12	渡し	渡す	VERB	動詞-一般	_	0	root	_	SpaceAfter=No|Translit=ワタシ
13	た	た	AUX	助動詞	_	12	aux	_	SpaceAfter=No|Translit=タ

>>> import deplacy
>>> deplacy.render(doc,Japanese=True)
太郎 PROPN ═╗<════════╗ nsubj(主語)
は   ADP   <╝         ║ case(格表示)
花子 PROPN ═╗<══╗     ║ nsubj(主語)
が   ADP   <╝   ║     ║ case(格表示)
読ん VERB  ═╗═╗═╝<╗   ║ acl(連体修飾節)
で   SCONJ <╝ ║   ║   ║ mark(標識)
いる AUX   <══╝   ║   ║ aux(動詞補助成分)
本   NOUN  ═╗═════╝<╗ ║ obj(目的語)
を   ADP   <╝       ║ ║ case(格表示)
次郎 PROPN ═╗<╗     ║ ║ obl(斜格補語)
に   ADP   <╝ ║     ║ ║ case(格表示)
渡し VERB  ═╗═╝═════╝═╝ ROOT(親)
た   AUX   <╝           aux(動詞補助成分)
>>> from deplacy.deprelja import deprelja
>>> for b in suparunidic.bunsetu_spans(doc):
...   for t in b.lefts:
...     print(suparunidic.bunsetu_span(t),"->",b,"("+deprelja[t.dep_]+")")
...
花子が -> 読んでいる (主語)
読んでいる -> 本を (連体修飾節)
太郎は -> 渡した (主語)
本を -> 渡した (目的語)
次郎に -> 渡した (斜格補語)

suparunidic.load(UniDic,BERT) loads a natural language processor pipeline, which uses UniDic for tokenizer POS-tagger and lemmatizer, then uses BERT for Biaffine dependency-parser of SuPar. Available UniDic options are:

• UniDic="gendai": Use 現代書き言葉UniDic
• UniDic="spoken": Use 現代話し言葉UniDic
• UniDic="novel": Use 近現代口語小説UniDic.
• UniDic="qkana": Use 旧仮名口語UniDic
• UniDic="kindai": Use 近代文語UniDic
• UniDic="kinsei": Use 近世江戸口語UniDic
• UniDic="kyogen": Use 中世口語UniDic
• UniDic="wakan": Use 中世文語UniDic
• UniDic="wabun": Use 中古和文UniDic
• UniDic="manyo": Use 上代語UniDic
• UniDic=None unidic-lite (default)

Available BERT options are:

• BERT="bert-japanese-aozora6m3m-unidic32k-2m" from bert-japanese-aozora (default)
• BERT="roberta-large-japanese-aozora" from roberta-large-japanese-aozora
• BERT="roberta-large-japanese-aozora-char" from roberta-large-japanese-aozora-char
• BERT="roberta-base-japanese-aozora" from roberta-base-japanese-aozora
• BERT="roberta-base-japanese-aozora-char" from roberta-base-japanese-aozora-char
• BERT="roberta-small-japanese-aozora" from roberta-small-japanese-aozora
• BERT="roberta-small-japanese-aozora-char" from roberta-small-japanese-aozora-char
• BERT="deberta-large-japanese-aozora" from deberta-large-japanese-aozora
• BERT="deberta-base-japanese-aozora" from deberta-base-japanese-aozora
• BERT="deberta-small-japanese-aozora" from deberta-small-japanese-aozora
• BERT="deberta-base-japanese-unidic" from deberta-base-japanese-aozora (fugashi required)
• BERT="bert-base-japanese-char-extended" from bert-base-japanese-char-extended
• BERT="bert-large-japanese-char-extended" from bert-large-japanese-char-extended
• BERT="bert-base-japanese-char" from cl-tohoku (fugashi and ipadic required)
• BERT="bert-base-japanese-whole-word-masking" from cl-tohoku (fugashi and ipadic required)
• BERT="bert-large-japanese" from cl-tohoku (fugashi required)
• BERT="bert-large-japanese-char" from cl-tohoku (fugashi required)
• BERT="roberta-base-japanese" from nlp-waseda (SentencePiece required)
• BERT="roberta-large-japanese" from nlp-waseda (SentencePiece required)
• BERT="electra-base-japanese-discriminator" from izumi-lab (fugashi and ipadic required)
• BERT="bert-small-japanese" from izumi-lab (fugashi and ipadic required)
• BERT="electra-base-japanese-generator" from izumi-lab (fugashi and ipadic required)
• BERT="japanese-roberta-base" from rinna (SentencePiece required)
• BERT="albert-japanese-v2" from alinear-corp (SentencePiece required)
• BERT="albert-base-japanese-v1" from ken11
• BERT="electra-small-japanese-discriminator" from Cinnamon AI
• BERT="electra-small-japanese-generator" from Cinnamon AI
• BERT="ku-bert-japanese-large" from ku-bert-japanese
• BERT="bert-base-ja-cased" from Geotrend
• BERT="laboro-bert-japanese-large" from Laboro AI
• BERT="nict-bert-base-japanese-100k" from NICT BERT
• BERT="unihanlm-base" from microsoft/unihanlm-base
• BERT="distilbert-base-japanese" from bandainamco-mirai

Installation for Linux

pip3 install suparunidic --user

Installation for Cygwin64

Make sure to get python37-devel python37-pip python37-cython python37-numpy python37-wheel gcc-g++ mingw64-x86_64-gcc-g++ git curl make cmake, and then:

curl -L https://raw.githubusercontent.com/KoichiYasuoka/CygTorch/master/installer/supar.sh | sh
pip3.7 install suparunidic

Benchmarks

Results of 舞姬/雪國/荒野より-Benchmarks

BERT="bert-japanese-aozora6m3m-unidic32k-2m"

BERT="roberta-large-japanese-aozora"

BERT="roberta-large-japanese-aozora-char"

BERT="roberta-base-japanese-aozora"

BERT="roberta-base-japanese-aozora-char"

BERT="roberta-small-japanese-aozora"

BERT="roberta-small-japanese-aozora-char"

BERT="deberta-large-japanese-aozora"

BERT="deberta-base-japanese-aozora"

BERT="deberta-small-japanese-aozora"

BERT="deberta-base-japanese-unidic"

BERT="bert-base-japanese-char-extended"

BERT="bert-large-japanese-char-extended"

BERT="bert-base-japanese-char"

BERT="bert-base-japanese-whole-word-masking"

BERT="bert-large-japanese"

BERT="bert-large-japanese-char"

BERT="roberta-base-japanese"

BERT="roberta-large-japanese"

BERT="electra-base-japanese-discriminator"

BERT="bert-small-japanese"

BERT="electra-base-japanese-generator"

BERT="japanese-roberta-base"

BERT="albert-japanese-v2"

BERT="albert-base-japanese-v1"

BERT="electra-small-japanese-discriminator"

BERT="electra-small-japanese-generator"

BERT="ku-bert-japanese-large"

BERT="bert-base-ja-cased"

BERT="laboro-bert-japanese-large"

BERT="nict-bert-base-japanese-100k"

BERT="unihanlm-base"

BERT="distilbert-base-japanese"

Author

Koichi Yasuoka (安岡孝一)