[PERF] tokenizer: work with slices

[LucasLefevre]

With this commit, we try to avoid building tokens character by character, but rather by slice of string where it makes sense.

Specifically, we try to avoid the pattern result += somethingin loops where something is a single character.

Total memory allocation of tokenize (BE timesheet dashboard)

before: 201Mb after: 128Mb (~36% less)

Total time to create the model:

before: 5003ms after: 4931ms (~70ms saved, ~1.5% faster) (average over 10 runs)

Description:

description of this task, what is implemented and why it is implemented that way.

Task: TASK_ID

review checklist

• [ ] feature is organized in plugin, or UI components
• [ ] support of duplicate sheet (deep copy)
• [ ] in model/core: ranges are Range object, and can be adapted (adaptRanges)
• [ ] in model/UI: ranges are strings (to show the user)
• [ ] undo-able commands (uses this.history.update)
• [ ] multiuser-able commands (has inverse commands and transformations where needed)
• [ ] new/updated/removed commands are documented
• [ ] exportable in excel
• [ ] translations (_t("qmsdf %s", abc))
• [ ] unit tested
• [ ] clean commented code
• [ ] track breaking changes
• [ ] doc is rebuild (npm run doc)
• [ ] status is correct in Odoo

[robodoo]