quickwit
quickwit copied to clipboard
Log tokenizer
Description
This pull request is just a draft in order to discuss the suggested fix for issue #1143.
How was this PR tested?
cargo check and cargo test has been run to check for regression + some added tests in quickwit-doc-mapper/tokenizer.rs
Please @nigel-andrews resolve the comments that you took into account. If relevant, it's nice to provide a feedback to the reviewer directly in the PR.