PSeitz

Hi, I took a closer look on the issue with some insight, although there are some things I don't completely understand with regard to the states data-structures. The issue seems...

Thanks for the links. I think I understood the basic concepts. Theses examples although all use characters in their states and not bytes, which makes them a lot easier, because...

I had another look on the issue, and the problem is the overwrite, where the old branch gets orphaned. That's the reason, why this appears only with same byte characters...

Thanks for the hint, that's pretty interesting. They seem to use two parameters ``, while copy_amount are 8, 16 and 32. `lz4_flex` uses a wild_copy of 16 currently, but depending...

Is it safe to read from multiple threads? (std::fs::File e.g. is not thread-safe)

There is a place where the next char is peeked: https://github.com/PSeitz/wana_kana_rust/blob/c45ebcdf3e8acc6421e967e792374ac390947df8/src/to_hiragana.rs#L29-L31 So it would require a peekable iterator I think https://doc.rust-lang.org/std/iter/struct.Peekable.html There may be other similar cases.

On second thought, we could always just use the minimal api instead of a uniform one, which would not require Peekable here. What do you think?

> > @PSeitz What do you mean by minimal vs. uniform API? > I mean defining a single iterator api which is used by all methods, vs minimal api everywhere.

Thanks for the suggestion, I think this API would be much better, than the existing one.

I don't think a single allocation per query won't have a measurable impact on performance, therefore I'd prefer the more readable variant.