Build linux wheels for typed_python

[changhiskhan]

Using manylinux we can build wheels across different python versions. This makes it easier to create binary distributions of typed_python so that it doesn't always need to be built from the source distro when installing.

More details are in the build_scripts/README.md attached in this PR.

I've tested this by installing the wheel into a conda environment and going through the typed_python

I haven't dug into the numpy version compat issues. @braxtonmckee would you happen to have examples I can use to try and repro any such issues? Potentially if we need an exact version match, I think we could generate the requirements file during the wheels build using the Pipfile.lock, so then setup.py can pick up the pinned requirements versions and install those exact versions.

We can do the same thing in object_database to build wheels and make the distro for both a lot simpler?

[changhiskhan]

not sure why the tests are failing -- I've not touched any of the existing code

[braxtonmckee]

Very cool! I'll take a look when I get a chance. On the numpy thing, here's how it works:

1. we have a dependency on numpy in typed python but we can use any version
2. you're supposed to depend in the pyproject.toml for the build on the "earliest installable version of numpy for this python" because if you build a C shared object against a specific version of numpy, it's binary compatible with forward versions, and this strategy ensures that when you build a version of the shared object you get one that will be compatible with any version of numpy that's installed on that target system.
3. unfortunately, the c++ code in object_database depends on typed python and must be compiled against the exact version of typed python that's currently installed. This goes against the assumptions that pip wants to make. I have it as a project to decouple ODB's dependency on TP to the point where we could be similarly "forward compatible" but have yet to do that

I don't know how the manylinux wheels works regarding numpy - if you build a wheel against the "earliest installable version of numpy for this python", that has to be a version that's determined, I guess, simply by the version of python you're on and the platform you're targeting. But I would imagine we can just do what other libraries that depend on numpy have done and it ought to work.

On Fri, Jul 29, 2022 at 3:31 PM Chang She @.***> wrote:

@changhiskhan https://github.com/changhiskhan requested your review on: #393 https://github.com/APrioriInvestments/typed_python/pull/393 Build linux wheels for typed_python.

— Reply to this email directly, view it on GitHub https://github.com/APrioriInvestments/typed_python/pull/393#event-7091721462, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB6OHBDVMJOJBCIEYFXAPEDVWQWRVANCNFSM55BUEQLA . You are receiving this because your review was requested.Message ID: @.*** com>

[changhiskhan]

Manylinux is agnostic to how you want to manage dependencies. What I've done here is that I'm using the Pipfile.lock for each python version to build to install the dependencies before building the corresponding wheel. Using pipenv these are all isolated in separate virtualenvs.

WRT object_database, we could do the same thing. And as long as we keep the typed_python versioned pinned in object_database requirements then if we installed the odb wheel it should be able to automatically grab the right wheel for typed_python if the wrong version is installed.

I wish there was an easier way to be macOS builds working properly. I don't want @dkrasner 's hair to get even crazier

[changhiskhan]

The Pipfile.lock all have different numpy versions but looking at them it doesn't seem like they're the oldest available. Should be a minor change though to get it using the build requires in pyproject.toml

[changhiskhan]

ok i switched to using python -m build which does respect the build dependencies specified in pyproject.toml.