Improve the integration with setuptools and the packaging ecosystem

[antocuni]

As described below, the current HPy/setuptools integration is a mess and could/should be vastly improved. And in general, we should try to engage more with the broader setuptools/pypa/packaging community. This issue wants to be a summary of the current status AND a set of open topics/questions that we can use as a starting point to talk to them.

Summary of current behavior of HPy+setuptools

From the user point of view:

• you use hpy_ext_modules=[...] instead of ext_modules=[...] in your setup.py

• the items of hpy_ext_modules are instances of the plain setuptools.Extension

• HPy modules can be compiled using two different ABIs: CPython ABI which produces e.g foo.cpython-38-x86_64-linux-gnu.so and HPy Universal ABI which produces foo.hpy.so (but we need a better extension -- see below)

• On CPython, the default is to use the CPython ABI. You can select the universal ABI by using the following option:

  $ python setup.py --hpy-abi=universal build
  

• other implementations are free to choose their own default ABI. E.g. on PyPy you get the universal ABI by default.

• when targeting the CPython ABI we produce a single file foo.cpython-38-x86_64-linux-gnu.so which is importable as-is by CPython.

• when targeting the HPy ABI we produce foo.hpy.so and foo.py: the latter contains some bootstrap code to actualy load the former.

Currently, we implemented the behavior described above by using a mixture of proper features and horrible monkey-patching (sorry for that). The relevant code is in hpy/devel/__init__.py: https://github.com/hpyproject/hpy/blob/1804201d316947a3c0dd4f1b2a2f7e14b976d303/hpy/devel/init.py

Notable implementation details:

hpy_ext_modules is implemented using an setuptools entry point: this is our entry point to do all the the rest: https://github.com/hpyproject/hpy/blob/1804201d316947a3c0dd4f1b2a2f7e14b976d303/hpy/devel/init.py#L133-L147

The horrible monkey patching happens here: https://github.com/hpyproject/hpy/blob/1804201d316947a3c0dd4f1b2a2f7e14b976d303/hpy/devel/init.py#L78-L117

Our moneky-patched build_ext uses finalize_options as a hook to be executed "at some point". In that hook, we take all the hpy_ext_modules, modify them in various ways and add them to the "normal" list of ext_modules: https://github.com/hpyproject/hpy/blob/1804201d316947a3c0dd4f1b2a2f7e14b976d303/hpy/devel/init.py#L308-L320

The Extension instances which were in hpy_ext_modules are modified in various ways: in particular we add the relevant include_dirs, sources and macros: https://github.com/hpyproject/hpy/blob/1804201d316947a3c0dd4f1b2a2f7e14b976d303/hpy/devel/init.py#L290-L306

In order to produce a binary whose extension is .hpy.so, we do another terrible hack: this is the code https://github.com/hpyproject/hpy/blob/1804201d316947a3c0dd4f1b2a2f7e14b976d303/hpy/devel/init.py#L330-L341

The horrible hack comes from the implementation of is_hpy_extension, because at this point in time hpy_ext_module=[...] and ext_modules=[...] are mixed together. The only way that we found was to wrap the extension name into a special subclass of str :facepalm:. We tried other ways (e.g. setting an is_hpy attribute on the extension itself) but they didn't work because the modifications are lost somewhere in the internals of distutils/setuptools: https://github.com/hpyproject/hpy/blob/1804201d316947a3c0dd4f1b2a2f7e14b976d303/hpy/devel/init.py#L181-L204

Writing the foo.py stub for universal mode was also tricky: https://github.com/hpyproject/hpy/blob/1804201d316947a3c0dd4f1b2a2f7e14b976d303/hpy/devel/init.py#L343-L386

And finally, we had to modify get_export_symbols in order to work on Windows: this is needed because the HPy universal ABI exports a function called HPyInit_*, not PyInit_*: https://github.com/hpyproject/hpy/blob/1804201d316947a3c0dd4f1b2a2f7e14b976d303/hpy/devel/init.py#L388-L398

---

Removing the hacks

We would like to work with the general community to find a set of hooks and features that will allow us to implement the desired functionality without using those horrible hacks, so we would like to input from the setuptools community and see what is their stance on it.

At the beginning I thought that we could propose a set of hooks to solve our needs, but then I realized that they surely have a better idea of what is doable, what is "good" and what is "bad", so probably it's better if they are the ones who propose the hooks/API/interface/whatever.

Note that from our point of view they end result doesn't necessarily need to be the very same as it is now. For example, currently we use hpy_ext_modules=..., but I would be very happy even with e.g. ext_modules=[Extension('foo', ..., hpy=True)] or someething like that, as long as setuptools calls us in a "clean" way.

---

Other build systems

I know that there are other build systems other than setuptools, but personally I don't know anything about them. I suppose we should have a story for people who wants to use HPy without setuptools: e.g. some better defined API to tell the world what are the include dirs, the extra sources, the resulting filename, etc. etc.

---

Filename extension

This is another open topic: currently we produce foo.hpy.so because it was the simplest thing to do, but eventually we would like to have official support by the community and the ecosystem. There are two easy thing which we can do now:

• add the CPU and platform tags to the filename

• add the HPy ABI version number to the filename

This would result in something like: foo.hpy-1-x86_64-linux-gnu.so.

Note for external readers: one cool thing about HPy is that the ABI is extensible by design, so once we finalize hpy-1 it is very likely that it will stat stable for years and years. But of course it is good to have a version number so that we can be ready if we will ever need to change something in the future.

---

PyPI/wheels support?

This is a direct consequence of the section above: what needs to happen to be able to have hpy wheels, uploaed them to PyPI and then pip install them?