NOKUBI Takatsugu

LIME tokenizer for SentencePiece (or other tokenizer)

3

Curently, it seems just use `str.split` so it didn't work with non-space segmented languages like Japanese. https://github.com/PAIR-code/lit/blob/3eb824b01e0f72a5486124b16056bf912465debc/lit_nlp/components/citrus/lime.py#L85 I tried to use it with SentencePiece-based model (japanese-ALBERT), but it handle input...

各種ディレクトリの作成について

1

ipynb内部で指定されたディレクトリが存在していることを前提としているため、いくつかの処理が失敗してしまいます。 ipynb中で ! mkdir -p data output あるいは os.makedirs("data", exist_ok=True) 等を追加するのはどうでしょうか。

I modified to work python 2/3 both. - use io.BytesIO on python3 - xrange as a alias of range on python3 - add parenthesis on print

When item list contains unicode string, map(unicode, item) raises UnicodeEncodeError exception. This patch fixes it.

The variable "it" in parsexml is a simple list object and not iteratable. So next(it) causes an error, the request fix it. I tested on python 2.7.9.

The repository lacks "model" directory, so I need to make it before training. Would you make the directory on the repository or mkdir code on train.py?