classifier-reborn In some languages like Chinese, a word of length not bigger than 2 is very common, so I suppose this is a very strong(sometimes wrong in other languages) assumption.

In some languages like Chinese, a word of length not bigger than 2 is very common, so I suppose this is a very strong(sometimes wrong in other languages) assumption.

Open Christophy opened this issue 6 years ago • 2 comments

https://github.com/jekyll/classifier-reborn/blob/4e807496e69cbb33ce2663564ef287f167915879/lib/classifier-reborn/extensions/hasher.rb#L30

Mar 03 '18 07:03 Christophy

We could probably make this configurable. I’ll happily review a PR for this.

Mar 03 '18 19:03 Ch4s3

@Christophy We just merged https://github.com/jekyll/classifier-reborn/pull/162, which allows for custom tokenizers. Could you let us know if this helps?

Mar 05 '18 18:03 Ch4s3

classifier-reborn classifier-reborn copied to clipboard

In some languages like Chinese, a word of length not bigger than 2 is very common, so I suppose this is a very strong(sometimes wrong in other languages) assumption.

classifier-reborn
classifier-reborn copied to clipboard