Argument clinic-like way to generate the argument parsing logic

[antocuni]

This is an attempt to start a discussion on how to implement function calls and function definitions. In the following, "functions" and "methods" are used interchangeably.

First, we need to identify who are the interested players and use cases:

• authors of C extensions who write C functions. Let's call them C-writers.

• Authors of C extensions who call Python functions. Let's call them C-callers.

• Generators of C extensions. Let's call them "cython" but it applies also to others.

• Python implementations with a JIT. Let's call them "pypy", but it applies also to others.

The basic idea is that we want to allow C-writers to write functions with a C-level signature, and generate the logic to do argument parsing automatically. E.g.:

long my_add_impl(long x, long y) { return x+y; };
AUTOMAGICALLY_GENERATE_PYTHON_FUNCTION("my_add", my_add_impl);

The precise details of how to implement AUTOMAGICALLY_GENERATE_PYTHON_FUNCTION are the scope of this discussion.

Goals & constraints

1. Writing functions in this way is optional. It will always be possible to write functions with the usual calling conventions such as HPyFunc_VARARGS and to the argument parsing manually.

2. There must be a way to quickly check whether a Python callable supports a given C-level signature and to get the underlying function pointer. "pypy" can use this to generate code which completely bypasses the python argument parsing logic, and cython" could use it to emit a fast-path in case it statically know the C types of the arguments.

3. Ideally, this should be integrated with the C API to call functions. E.g., if a C-caller calls HPy_Call(my_add, "ll", 4, 5), an implementation should be able to bypass argument parsing and call my_add_impl directly.

4. Bonus point if we find a way to implement goal 3 also on CPython.

5. It should be possible to do things manually: i.e., a C-writer could write its own argument parsing code and be able to declare a C-level signature. In that case, he needs to ensure that its own argument parsing code does the "correct things". In particular, when you do call from Python like my_add(4, 5), the implementation should be free to decide whether to call the generic version or the C-specialized overload.

6. To be discussed: we need to decide whether we want to support "overloads" or not. I.e., a given Python functions could in principle support many different C-level signatures. I think that all the following options are reasonable:

   • support at most one C-level signature. Functions which can't be encoded this way needs to be written in the "old style" and parse arguments manually.

   • Support at most N C-level signatures, where N is a small number like 4. If you require more than N, you have to write parsing manually.

   • Support a potentially unlimited number of C-level signatures.

Argument Clinic vs C macros

CPython has already something similar: it is called Argument Clinic and AFAIK it's used only internally. You have to write special comments in the C code to specify the signature of your function, then you run a python script which edits the C files and adds the relevant autogenerated code. In the following, when I talk about "Argument Clinic" I don't necessarily mean the very same clinic.py as CPython. We will probably have to write our own version, but the concept is the same.

On the other hand, HPy so far has relied on macros to generate pieces of code. Consider the following example:

HPyDef_METH(double_obj, "double", double_obj_impl, HPyFunc_O)
static HPy double_obj_impl(HPyContext ctx, HPy self, HPy obj)
{
    return HPy_Add(ctx, obj, obj);
}

Here, HPyDef_METH is a macro which among the other things generates a small trampoline to convert from the CPython calling convention to the HPy calling convention. It generates something similar to this (in the CPython-ABI case):

static PyObject *
double_obj(PyObject *self, PyObject *arg)
{
    return _h2py(double_obj_impl(_HPyGetContext(), _py2h(self), _py2h(arg)));
}

So one option is to extend this functionality to generate also the argument parsing logic (more on that later).

Both options have pros and cons, IMHO:

• Clinic PRO: is already known by CPython devs and it seems to work well.

• Clinic PRO (very futuristic): it will be easier for CPython itself to migrate to HPy.

• Clinic CON: C-writers might dislike the fact that an external script modifies their C source, potentially cluttering them with lines and lines of obscure code. It might be possible to put all the generated code into a separate file though.

• Macros PRO: they just work with any C compiler. The C code which you have to write is probably more compact and/or nicer to read.

• Macros CON: probably we will be more limited in the complexity of logic we can generate (maybe it's a PRO :)).

• Macros CON: compiler erros are potentially more obscure, although so far in HPy we managed to get very good compiler errors in response to common mistakes, at least with gcc.

• Macros CON: we need to put an upper bound to the number of arguments, because we need to autogen a file which contains the macros for all possible C signatures. We also need to check whether this impacts compilation time negatively.

How to encode C signatures

There are at least two ways to encode/specify C signatures:

• use a C string: this is more or less equivalent to what you can pass to HPyArg_Parse, although we need to extend the notation to specify the return type. E.g., "d ll" could mean double _(long, long).

• extend the enum HPyFunc_Signature to support many more signatures. So, in addition to HPyFunc_VARARGS, HPyFunc_NOARGS etc., you have e.g. HPyFunc_d_ll which corresponds to double _(long, long). In this scenario each signature is represented by a single int64_t value, and there are at least a couple of variations for how to encode it:

  • if we decide that we are happy to support only long, double, HPy and void, we can encode a single type in 2 bits. So, we can specify signatures up to 31 arguments (2 bits are reserved for the return type), maybe a bit less if we want to save some bits to encode other interesting features (e.g. if the function supports varargs or keywords).

  • Use 8 bits for each param: with this we can support many more types, but we are limited to signatures up to ~7 arguments, maybe 6 if we want to reserve some bits for other features. 6-7 arguments are enough to cover the vast majority of functions though: if a function wants to use more args, it has to do argument parsing by itself.

Pros/cons of each approach:

• C string PRO: easy to understand, very flexible.

• C string CON: it's impossible to do any compile-time type check.

• C string CON: I think it's impossible to implement it with macros. The current approach of using HPyFunc_* works because we can write "specialized" versions of HPyFunc_TRAMPOLINE for each possible value of HPyFunc_Signature, but I have no clue how to do it with a generic C string using macros. So, if we choose this, we are automatically choosing argument clinic.

• HPyFunc PRO: checking whether a callable supports a given signature is very quick, since you just compare two ints. Do the same check with strings is probably slower because you need a strcmp.

• HPyFunc PRO: works with the macros approach.

• HPyFunc PRO: we can probably write something which does compile-time checks of the argument types.

• HPyFunc CON: much less flexible that C strings. The C syntax for calling is probably also less nice, e.g. HPy_Call(HPyFunc_d_ll, 4, 5) vs HPy_Call("ll", 4, 5).

• HPyFunc CON: if we use the macros approach, we probably need to generate a huge header with all the macros definitions, which might impact the build time.

We can also adopt a hybrid approach: the user-facing API takes and receives C strings for signatures, but internally we represent them inside an encoded int64_t. This should make runtime signature checks faster (but it's probably a good idea to do some benchmarks).

Return types

Another open question is what to do with return types. Consider the example above in which I have the function my_add whose signature is "d ll" (i.e., double _(long, long)):

HPy res = HPy_Call("ll", my_add, 4, 5);

in this case, the return type is HPy. But what if I want to call the C function directly and the a double back, without having to box it? Cython surely needs an API to do that. So maybe something like this:

double result;
void *fnptr = HPyFunc_Try("d ll", my_add);
if (fnptr)
    result = fnptr(4, 5);
else
    result = HPyFloat_AsDouble(HPy_Call("ll", my_add, 4, 5));

Suggestions for a better name instead of HPyFunc_Try are welcome.

Runtime signature checks

Why do we need fast runtime signature checks? I can think of at least two use cases:

• HPy_Call: you can add a fast-path: if the callable supports the given signature, you call it directly, bypassing the boxing/unboxing. But it is unclear whether it is doable since HPy_Call knows only the types of the arguments, not the type of the result.

• HPyFunc_Try: see above, this is needed by Cython.

Note that this doesn't apply to "pypy": assuming that the callee is known, the JIT can do the signature check at compile time, so it doesn't have to be particularly efficient.

Relationship with the current HPyFunc_*

Currently the enum HPyFunc_Signature defines ~30 signatures which are used by methods and slots. We need to understand whether the represent the same thing as the C-level function signatures or whether they are completely different beasts. E.g., HPyFunc_O is basically equivalent to "O O", HPyFunc_BINARYFUNC to "O OO", HPyFunc_INQUIRY to "i O", etc.

Proposal #1: Argument clinic and C strings

Let's try to turn this into something more concrete. At the moment, I am not happy with eiher of those though. The following is a sketch proposal which uses the "Argument clinic" and "C string" approaches described above:

/*[hpy-clinic input]
my_add

  return: "d"
  a: "l"
  b: "l"
  
add two numbers together
 [hpy-clinic start generated code]*/
... code generated by hpy-clinic ...
/*[hpy-clinic end generated code]*/

static double my_add_impl(HPyContext ctx, long a, long b)
{
    return (double)(a+b);
}

What I don't like too much of this approach is that it's completely different that the HPyDef_METH that you use for non-argument clinic methods. Maybe something like this, in which we put the generated code BEFORE the call to HPyDef_METH_CLINIC? But note that in this way you loose the names of the arguments:

/*[hpy-clinic start generated code]*/
...
/*[hpy-clinic end generated code]*/
HPyDef_METH_CLINIC(my_add, "my_add", my_add_impl, "d ll")
static double my_add_impl(HPyContext ctx, long a, long b)
{
    return (double)(a+b);
}

Proposal #2: HPyFunc_* and macros

The following integrates very well with the existing API, with all the pros&cons described in the sections above.

HPyDef_METH(my_add, "my_add", my_add_impl, HPyFunc_d_ll)
static double my_add_impl(HPyContext ctx, long a, long b)
{
    return (double)(a+b);
}

EDIT: s/4 bits/2 bits in the "How to encode C signatures" section