add support for multiple Python3 versions (not just 3.7)

[christian-korneck]

Currently only Python 3.7 is supported. This is because the Python3 C API slightly changes between versions. For example Python 3.8 removed the PyEval_ReInitThreads function, so binding it fails. (Also sometimes new functions get added).

We should add support for multiple Python versions. This should happen in a way that we don't just delete the functions that aren't supported in newer Python versions anymore (as this would be a breaking change for users of older Python versions).

If breaking changes are unavoidable, this should lead to a new major version.

[lsm1103]

Whether you can use the factory class design pattern to make it compatible with existing Python versions to add functionality to Python 3.8 and above

[christian-korneck]

I’ve been thinking about two “UX” options so far:

• a factory function that can be used like

python311 := cpy3.NewPython("3.11")
python311.PyInitialize()

• multiple package imports, most likely with a split between the Python3 core API (which stays stable for all of Python3) and a version specific package on top.

import (
	py3core "github.com/go-python/cpy3/core"
	py311 "github.com/go-python/cpy3/3-11"

)

What are your thoughts and preferences? Would appreciate any suggestions and opinions from everyone. Thanks!!

[nathantfrank]

I think the second option of multiple package imports leans closer to pragmatic go.

[jshiwam]

Is anyone working on this issue? I want to take this up.

[christian-korneck]

@jshiwam I’m only doing experiments so far. Feel free to open a PR to collaborate.

[pwaller]

I don't see how you could do the cpy3.NewPython("3.11") without linking in lots of cpython versions, unless I'm missing something. I'm not sure i'd want more than one python linked in! I think it would have to be separate import paths.

(And also a big +1, I now need python3.8 and I'll want more in the future!)

[jshiwam]

@pwaller Here only one version of cpython would be linked. My understanding is that the Core API would be implemented by the parent, and the child classes will have the implementation of specific versions. If you want to use python3.11 then you would do cpy3.NewPython("3.11")

[jshiwam]

@pwaller @christian-korneck here is a POC that I have created. Please take a look at it. This POC takes the second approach of having submodules for different versions rather than Using a factory function.

https://github.com/jshiwam/cpy3x

I have implemented really few apis currently. Let me know if this looks good, and could be developed into something that we have suggested in this issue.

I have tried linking multiple interpreters and it seems to work too. Here is the commit : https://github.com/jshiwam/cpy3x/commit/d5263f6c0cc096433781ae7e97969f2e405f7a8a

[jshiwam]

Hi Everyone, I am building the generic version of this library for python3.

//CallFunctionObjArgs : https://docs.python.org/3/c-api/object.html#c.PyObject_CallFunctionObjArgs
func (pyObject *PyObject) CallFunctionObjArgs(args ...*PyObject) *PyObject {

	if len(args) > MaxVariadicLength {
		panic("CallFunctionObjArgs: too many arrguments")
	}
	if len(args) == 0 {
		return togo(C._go_PyObject_CallFunctionObjArgs(toc(pyObject), 0, (**C.PyObject)(nil)))
	}

	cargs := make([]*C.PyObject, len(args), len(args))
	for i, arg := range args {
		cargs[i] = toc(arg)
	}
	return togo(C._go_PyObject_CallFunctionObjArgs(toc(pyObject), C.int(len(args)), (**C.PyObject)(unsafe.Pointer(&cargs[0]))))
}

I don't understand why we have used MaxVariadicLength, the cpy3/object.go file has this comment which says the following:

//MaxVariadicLength is the maximum number of arguments that can be passed to a variadic C function due to a cgo limitation
const MaxVariadicLength = 20

I just wrote a sample program to check if that's the case currently or not. The following CGO program can take more than 22 arguments.

#include<stdio.h>

int Area(int a, int b, int c, int d, int e, int f, int g,int h, int i, int j, int k, int l, int m, int n, int o, int p, int q, int r, int s, int t, int u, int v){
    return a+b+c+d+e+f+g+h+i+j+k+l+m+n+o+p+q+r+s+t+u+v;
}

And it works without any issues. Was it a limitation before when this library was built? Should i change it in the new version?

[M-Quadra]

I tend to use

github.com/go-python/cpy3/v8
github.com/go-python/cpy3/v9

for 3.8 or 3.9

[christian-korneck]

Thanks everyone! From my view also code generation is on the table.

I hope to do some tests and ideally make a decision and lay out what help we might need early January.

[darshan-verkada]

@christian-korneck any update on Python 3.8 support ?

[WillMeng]

any progress? @christian-korneck

[christian-korneck]

not at this time, apologies. I’m exceptionally busy with something else. However, to avoid misunderstandings, this project is not abandoned and I’m looking forward to make this change.