General CI refresh

[crusaderky]

This PR is propaedeutic to #1110.

Global changes

• Upgrade away from ubuntu-20.04 and actions/upload-artifact@v3, which have both been browned out.
• GitHub action artifacts from matrix-parameterised jobs no longer overwrite each other.
• Version bump of all github actions .
• Enable "Re-run all/failed jobs" button in GitHub Actions UI.

C++ changes

• Release C++ binaries for Linux ARM and Windows ARM.
• Run C++ unit tests on Windows ARM.
• Run C++ unit tests on Linux ARM natively (before, they were only running on QEMU).

Python changes

• Formally state that the minimum supported version of Python is 3.6.
• Run unit tests on Python 3.6 through to 3.13, whereas before they ran exclusively on 3.13, for:
  • Linux x64
  • Windows x86
  • Windows x64
• Run unit tests on Python 3.6 through to 3.13, whereas before they did not run at all, for:
  • Linux i686
  • Linux ARM
  • Windows ARM
• Move the execution of the unit tests from actions/setup-python to inside the cibuildwheels containers. This was necessary to cover Python 3.6, which is no longer deployable with actions/setup-python. Python 3.7 would have required pinning ubuntu-22.04. This also greatly simplifies the CI workflow under the expectation that unit tests must run against multiple versions of Python.
• Unit tests are now executed downstream of wheels creation. This means that CI is now checking that repair-wheels / auditwheels etc. work and that the Python wheels are actually self-contained.
• Build Linux ARM wheels natively instead of on QEMU for performance.
• Run twine check on the sdist archive.
• Migrate from python setup.py to python -m build.
• Rebuild pip lockfile; bump up all versions.

[crusaderky]

@taku910 could you enable CI please?

[crusaderky]

@taku910 this is ready for review. CI is green on my fork (https://github.com/crusaderky/sentencepiece/pull/1).

[crusaderky]

@taku910 gentle ping

[leondz]

@taku910 @around1991 Could we have this merged and also a PyPI release, to enable sentencepiece to continue working for the current and future Python releases?

[reneleonhardt]

Wow, awesome modernization work 🚀 But it looks like no maintainer is around anymore, I wouldn't expect a reaction anytime soon...

[alberthli]

Hi, any updates on this? Would be really helpful to be able to install sentencepiece on python 3.13 environments!

[chenyenru]

Would be really helpful to have this PR merged for Python 3.13 pypi support!

[BLKSerene]

Great, any updates on this?

[reneleonhardt]

@crusaderky Can you request another reviewer? I'm no maintainer 😉

Also bad that CI is not allowed to run automatically in Pull Requests... makes contributing even harder. This project sure feels abandoned, 16 months without a new release in the AI ecosystem is an achievement on it's own 😅

Begs the question why a state-of-the-art AI reviewer like CodeRabbit hasn't been enabled here to speed-up contributions? 🤔

[crusaderky]

@taku910 gentle ping

[SaberaTalukder]

This would be super helpful!

[ivandariojr]

Please merge this, our project cannot upgrade to python 3.13 solely because of this dependency.

[reneleonhardt]

While everyone keeps waiting for this project to resume maintenance (no release since 64 commits) and Python 3.14 coming out soon you could switch to a Python 3.13 build a community member contributed, thank you @dbowring! https://github.com/google/sentencepiece/pull/1084#issuecomment-2683926716 https://pypi.org/project/dbowring-sentencepiece/

[crusaderky]

superseded by master.