Proposal: Add Golang Bindings for tokenizers

[Nav31]

• Performance-Critical Applications – Many performance-intensive apps use Go for concurrency and efficiency. Native bindings allow seamless integration without overhead.
• Server-Side NLP – Go is widely used in backend services. Golang bindings would make it easier to use Hugging Face tokenizers in APIs, microservices, and inference systems.
• Better Compatibility – Many Go-based projects rely on Python/C++ workarounds for NLP tasks. Direct bindings would reduce complexity and improve developer experience.
• Improved Inference Pipelines – Many real-time AI systems are built in Go. Native tokenizers would remove Python bottlenecks, making pipelines faster and more reliable

[strategy155]

How do you imagine that? extern and cgo combination?

Or something more high level, like uniffi-rs, rust2go or fcplug?

[ArthurZucker]

Wow ahha that's a lot of reactions!

[ArthurZucker]

we unfortunately don't have the man power to maintain this 😿