Is it possible to pass a tokenizer from Python into Rust?

[albertsgarde]

I am implementing a part of my Python project in Rust. I need to use a tokenizer deep in the Rust code, but I want the Python code to decide what tokenizer to use. I would therefore like to pass a tokenizer from Python to Rust. Since the Python wrapper is supposed to be thin, I have hope that this is possible. If it is not, I'll probably just use Pyo3 to call Python from Rust, but that would both be uglier and feel silly to call Python code that only calls right back into Rust.

What I have tried looks something like this:

use tokenizers::Tokenizer;

#[pyfunction]
pub fn foo(tokenizer: &Tokenizer) {
	baz(tokenizer)
}

but Pyo3 gives me the error "the trait bound &tokenizers::Tokenizer: pyo3::PyClass is not satisfied". Looking at the source, this makes sense since it is the PyTokenizer struct in the crate tokenizers-python that wraps Tokenizer and is turned into a Python class. If I could access the PyTokenizer struct in Rust I might therefore be able to fix the problem, but since tokenizers-python is not available on crates.io this is difficult.

Is there some way to pass a tokenizer from Python to Rust?

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[ArthurZucker]

SOrry for the late reply here, I think pyo3 is required for this TBH. We have example of this, we use wrappers and match types to switch between model or decoder objects.