jonvet

I'm struggling to get a text representation **including formatting** from a `YText`. There is `to_json` but that just returns the raw text without formatting. Is there a plan to implement...

Fixed Length Pre-Tokenizer

3

Introduces a pre-tokenizer to split text in fixed length chunks (closes https://github.com/huggingface/tokenizers/issues/1697). The method `pre_tokenize` could be more made more concise by creating a vector with indices first like so...

fix: allow for self-referencing pydantic schema

10

## Describe your changes This PR introduces a new field `related_products` in the `Products` fixture which replicates the failure case mentioned in #155 and makes existing tests fail with a...