BorisTheBrave issues

Repositories
Issues
Comments

Results 11 issues of


                                            BorisTheBrave

[Bug Report] tokenize_and_concatenate doesn't tokenize correctly.

**Describe the bug** tokenize_and_concatenate slices strings into 20 chunks by character before tokenizing. This can cut a token in two, leading to token pairs that would never normally occur. **Code...