Separate python bindings
Repository needs to be restructured and python bindings to be separated from the core rust interfaces, which are currently all mixed and tied together, difficult to extend and support and make separate decisions about them.
Python bindings need to be in new bindings/python directory with their own set of dependencies and relevant python logic and tooling, with their own Cargo.*, .gitignore, pyproject, setup.*, Makefile, etc. Since python bindings can be considered as a built artifact, we also need to consider keeping Cargo.lock in git for reproducible builds. If this will be decided upon, then we would want to make certain adjustments in github workflows to test outside locked dependencies too and maybe add some other adjustments: https://github.com/dottxt-ai/outlines-core/issues/139#issuecomment-2612522589
To achieve that:
- Logic in core rust interfaces and in python bindings needs to be adapted
- Python benchmarks and tests moved too
- Interfaces of python bindings split into
src:guide.rs,index.rs,vocabulary.rsfiles, with python interfaces inpy_src/outlines_core - Benchmarks, tests, all github workflows updated and tested
The 1)-3) work was attempted to be done as part of this PR: https://github.com/dottxt-ai/outlines-core/pull/143 But it requires a lot of adjustments and many separate improvements on 4), that it wasn't finished and could be considered a showcase of direction: https://github.com/dottxt-ai/outlines-core/tree/python-bindings-separation-example-do-not-delete
However, there is a lot of work needs to be done in 4) and to make it easier, I would suggest, that these issues will be tackled first:
- https://github.com/dottxt-ai/outlines-core/issues/105
- simplify python related tooling and improve style checks with
ruffinstead