outlines-core icon indicating copy to clipboard operation
outlines-core copied to clipboard

Investigate serialization solution after python bindings stabilization

Open torymur opened this issue 1 year ago • 0 comments

We need to investigate serialization options after python bindings will stabilize (and loose many intermediary structures we have to support right now).

One solution was introduced with bincode and Python::with_gil, which is currently considered to be a temp measure to prevent vLLM integration failures: https://github.com/dottxt-ai/outlines-core/pull/99

There are some other potential issues with bincode, which needs to be confirmed. For example, it's not self-descriptive, so if Index changes in any way (e.g., field order, type, or size), deserialization will fail. Also bincode serialization may produce architecture-dependent binaries, differences in endianness or data alignment between 32-bit and 64-bit systems could lead to deserialization errors. bincode has support for serde integration, which could be enabled with the serde feature (we'd need to check for the exact impl details too).

torymur avatar Dec 02 '24 19:12 torymur