Bump tokenizers from 0.15.2 to 0.19.1

[dependabot[bot]]

Bumps tokenizers from 0.15.2 to 0.19.1.

Release notes

Sourced from tokenizers's releases.

v0.19.1

What's Changed

• add serialization for ignore_merges by @​ArthurZucker in huggingface/tokenizers#1504

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.19.0...v0.19.1

v0.19.0

What's Changed

• chore: Remove CLI - this was originally intended for local development by @​bryantbiggs in huggingface/tokenizers#1442
• [remove black] And use ruff by @​ArthurZucker in huggingface/tokenizers#1436
• Bump ip from 2.0.0 to 2.0.1 in /bindings/node by @​dependabot in huggingface/tokenizers#1456
• Added ability to inspect a 'Sequence' decoder and the AddedVocabulary. by @​eaplatanios in huggingface/tokenizers#1443
• 🚨🚨 BREAKING CHANGE 🚨🚨: (add_prefix_space dropped everything is using prepend_scheme enum instead) Refactor metaspace by @​ArthurZucker in huggingface/tokenizers#1476
• Add more support for tiktoken based tokenizers by @​ArthurZucker in huggingface/tokenizers#1493
• PyO3 0.21. by @​Narsil in huggingface/tokenizers#1494
• Remove 3.13 (potential undefined behavior.) by @​Narsil in huggingface/tokenizers#1497
• Bumping all versions 3 times (ty transformers :) ) by @​Narsil in huggingface/tokenizers#1498
• Fixing doc. by @​Narsil in huggingface/tokenizers#1499

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.15.2...v0.19.0

v0.19.0rc0

Bumping 3 versions because of this: https://github.com/huggingface/transformers/blob/60dea593edd0b94ee15dc3917900b26e3acfbbee/setup.py#L177

What's Changed

• chore: Remove CLI - this was originally intended for local development by @​bryantbiggs in huggingface/tokenizers#1442
• [remove black] And use ruff by @​ArthurZucker in huggingface/tokenizers#1436
• Bump ip from 2.0.0 to 2.0.1 in /bindings/node by @​dependabot in huggingface/tokenizers#1456
• Added ability to inspect a 'Sequence' decoder and the AddedVocabulary. by @​eaplatanios in huggingface/tokenizers#1443
• 🚨🚨 BREAKING CHANGE 🚨🚨: (add_prefix_space dropped everything is using prepend_scheme enum instead) Refactor metaspace by @​ArthurZucker in huggingface/tokenizers#1476
• Add more support for tiktoken based tokenizers by @​ArthurZucker in huggingface/tokenizers#1493
• PyO3 0.21. by @​Narsil in huggingface/tokenizers#1494
• Remove 3.13 (potential undefined behavior.) by @​Narsil in huggingface/tokenizers#1497
• Bumping all versions 3 times (ty transformers :) ) by @​Narsil in huggingface/tokenizers#1498

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.15.2...v0.19.0rc0

Commits

• 3b3c960 version 0.19.1
• da30768 "0.19.1-rc0" and Cargo.lock
• 7733bc2 add serialization for ignore_merges (#1504)
• 91393ef Fixing doc. (#1499)
• 949d9e3 Bumping all versions 3 times (ty transformers :) ) (#1498)
• e0defa7 Remove 3.13 (potential undefined behavior.) (#1497)
• d5a8cc7 PyO3 0.21. (#1494)
• 914576f Add more support for tiktoken based tokenizers (#1493)
• 6e58f83 version = "0.16.0-dev.0"
• 0906971 Refactor metaspace (#1476)
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

[github-actions[bot]]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

Package	Version	Score	Details
pip/tokenizers	0.19.1	:green_circle: 5	Details Check Score Reason Code-Review :green_circle: 6 Found 18/28 approved changesets -- score normalized to 6 Maintained :green_circle: 10 20 commit(s) and 5 issue activity found in the last 90 days -- score normalized to 10 CII-Best-Practices :warning: 0 no effort to earn an OpenSSF best practices badge detected License :green_circle: 10 license file detected Branch-Protection :warning: -1 internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration Signed-Releases :warning: 0 Project has not signed or included provenance with any releases. Binary-Artifacts :green_circle: 10 no binaries found in the repo Dangerous-Workflow :green_circle: 10 no dangerous workflow patterns detected Security-Policy :warning: 0 security policy file not detected Token-Permissions :warning: 0 detected GitHub workflow tokens with excessive permissions Fuzzing :warning: 0 project is not fuzzed Packaging :green_circle: 10 packaging workflow detected SAST :warning: 0 SAST tool is not run on all commits -- score normalized to 0 Vulnerabilities :green_circle: 7 3 existing vulnerabilities detected Pinned-Dependencies :warning: 0 dependency not pinned by hash detected -- score normalized to 0
pip/tokenizers	0.15.2	:green_circle: 5	Details Check Score Reason Code-Review :green_circle: 6 Found 18/28 approved changesets -- score normalized to 6 Maintained :green_circle: 10 20 commit(s) and 5 issue activity found in the last 90 days -- score normalized to 10 CII-Best-Practices :warning: 0 no effort to earn an OpenSSF best practices badge detected License :green_circle: 10 license file detected Branch-Protection :warning: -1 internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration Signed-Releases :warning: 0 Project has not signed or included provenance with any releases. Binary-Artifacts :green_circle: 10 no binaries found in the repo Dangerous-Workflow :green_circle: 10 no dangerous workflow patterns detected Security-Policy :warning: 0 security policy file not detected Token-Permissions :warning: 0 detected GitHub workflow tokens with excessive permissions Fuzzing :warning: 0 project is not fuzzed Packaging :green_circle: 10 packaging workflow detected SAST :warning: 0 SAST tool is not run on all commits -- score normalized to 0 Vulnerabilities :green_circle: 7 3 existing vulnerabilities detected Pinned-Dependencies :warning: 0 dependency not pinned by hash detected -- score normalized to 0

Scanned Manifest Files

setup.py

• [email protected]
• [email protected]