sentencepiece
sentencepiece copied to clipboard
google
Unsupervised text tokenizer for Neural Network-based text generation.
As the title says Tried all versions from 0.1.95 to 0.2.0 Fixed it by using python 3.11 instead Error log on python 3.13 and sentencepiece 0.2.0: ``` (venv) PS D:\ComfyUI>...
1. Fix with gcc 15 (include cstdint) 2. Release a new version and upload to Pypi, so that CMake 4 works
In the arguments of sentence piece tokemizer, the type of character coverage is int. However, it accepts a value b/w 0.98 and 1 ```` from speechbrain.tokenizers.SentencePiece import SentencePiece save_folder='./' output_neurons=60...
## Background OpenHarmony is an open-source operating system and community that is widely used in the field of intelligent terminals. Sentencepiece for unsupervised text tokenizer for neural network based text...
### What is the current behavior? Unable to install the package from pip. ### What is the desired behavior? To have [native wheel for WoA](https://blogs.windows.com/windowsdeveloper/2025/04/14/github-actions-now-supports-windows-on-arm-runners-for-all-public-repos/). GitHub Actions now supports win-arm64...
See #1110 This PR enables support for free-threaded CPython 3.13, most of the changes are related to testing, which enable parallel testing using pytest-run-parallel. Additionally, ThreadSanitizer checks were performed without...
Bumps the github-actions group with 6 updates in the / directory: | Package | From | To | | --- | --- | --- | | [actions/upload-artifact](https://github.com/actions/upload-artifact) | `3` |...
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot] avatar"){kind=avatar}
Bumps the build-time-deps group with 6 updates in the /.github/workflows/requirements directory: | Package | From | To | | --- | --- | --- | | [cibuildwheel](https://github.com/pypa/cibuildwheel) | `2.20.0` |...
This replaces `third_party/absl/` and `third_party/protobuf-lite/` with git submodules checked out at specific versions for `abseil-cpp`, `googletest`, `benchmark`, and `protobuf` libraries. This change allows removing local code copies of absl and...
As some of you might be aware, since CPython 3.13, there is experimental support for an alternative interpreter distribution (via [PEP703](https://peps.python.org/pep-0703/)) that disables the GIL altogether, thus enabling support for...