Breaking change in 0.21.2

[phil-opp]

There was a breaking change to tokenizers::normalizers::utils::Sequence in v0.21.2. In v0.21.1, the struct had a get_normalizers method, which was removed in v0.21.2.

This breaks the build of mistral-rs v0.5.0, as cargo considers tokenizers v0.21.4 and v0.21.2 as semver-compatible to v0.21.1. The workaround is to manually downgrade the tokenizers dependency in the Cargo.lock using cargo update tokenizers --precise 0.21.1, but this will be undone by the next cargo update.

The proper fix would be to do a v0.21.5 release that brings those methods back.

[phil-opp]

Given that there was another semver-incompatible breaking change recently in #1815, it might make sense to run cargo-semver-checks on the CI.

[phil-opp]

See also https://github.com/EricLBuehler/mistral.rs/issues/1523

[ArthurZucker]

This does make sense, will have a look at cargo-semver-checks