go-python3 python3.PyImport_ImportModule(name) will emit a fatal error when called the second time.

Describe what happened: Environments: On MacOS (Catalina Version 10.15.4) Python3.7.6 Go1.13.8 I want to use go-python3 to invoke an algorithm written in Python3, but as described, an fatal error will generated when the second time I invoke this algorithm. From the output message, it seems that PyImport_ImportModule causes this error.

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xa pc=0x91256a3]

runtime stack:
runtime.throw(0x4967a75, 0x2a)
        /usr/local/go/src/runtime/panic.go:774 +0x72
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:378 +0x47c

goroutine 41 [syscall]:
runtime.cgocall(0x4637740, 0xc000063c18, 0x48a4660)
        /usr/local/go/src/runtime/cgocall.go:128 +0x5b fp=0xc000063be8 sp=0xc000063bb0 pc=0x4004d0b
github.com/DataDog/go-python3._Cfunc_PyImport_ImportModule(0x8061d90, 0x0)
        _cgo_gotypes.go:3780 +0x4a fp=0xc000063c18 sp=0xc000063be8 pc=0x462c2fa
github.com/DataDog/go-python3.PyImport_ImportModule(0x49501f5, 0x8, 0x0)
        /Users/zhao/go/pkg/mod/github.com/!data!dog/[email protected]/import.go:24 +0x87 fp=0xc000063c80 sp=0xc000063c18 pc=0x462e267
PPGServer/pkg/algo.ImportModule(0x4964926, 0x26, 0x49501f5, 0x8, 0x1)
        /Users/zhao/go/src/PPGServer/pkg/algo/ppg.go:42 +0x4cb fp=0xc000063d98 sp=0xc000063c80 pc=0x46332db
PPGServer/pkg/algo.CalcPre(0xc0003560c0, 0xd, 0x0, 0x0)
....

Here is the sample code. A wrapper of PyImport_ImportModule:

// ImportModule will import python module from given directory
func ImportModule(dir, name string) *python3.PyObject {
	fmt.Println("python3.PyImport_ImportModule before")
	sysModule := python3.PyImport_ImportModule("sys") // import sys
	fmt.Println("python3.PyImport_ImportModule success")
	path := sysModule.GetAttrString("path")           // path = sys.path
	ob := python3.PyList_GetItem(path, 1)
	fmt.Println("check:", python3.PyUnicode_Check(ob))
	fmt.Println("path:", python3.PyUnicode_AsUTF8(ob))
	fmt.Println("sysModule.GetAttrString success")
	python3.PyList_Insert(path, 0, python3.PyUnicode_FromString("/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages"))
	python3.PyList_Insert(path, 0, python3.PyUnicode_FromString(dir))
	fmt.Println("After module insert:", python3.PyUnicode_AsUTF8(python3.PyList_GetItem(path, 0)))
	fmt.Println("module name:", name)
	return python3.PyImport_ImportModule(name) 
}

Each time the algorithm is called in a goroutine.

func CalcPre(dataFilePath string) (sbpI int, dbpI int) {
	python3.Py_Initialize()
	if !python3.Py_IsInitialized() {
		fmt.Println("Error initializing the python interpreter")
		os.Exit(1)
	}
	gstate = python3.PyGILState_Ensure()
	fmt.Println("Py_Initialize success")
	vbp := ImportModule("/Users/zhao/Desktop/lab/ppython", "value_bp")
	fmt.Println("ImportModule success")

	b := vbp.GetAttrString("estimate")
	fmt.Printf("[FUNC] b = %#v\n", b)
	bArgs := python3.PyTuple_New(1)
	python3.PyTuple_SetItem(bArgs, 0, python3.PyUnicode_FromString(dataFilePath))
	re := b.Call(bArgs, python3.Py_None)
	sbp := python3.PyTuple_GetItem(re, 0)
	dbp := python3.PyTuple_GetItem(re, 1)
	defer func() {
		python3.Py_Finalize()
		fmt.Println("python3.Py_Finalize()")
	}()
	sbpI = python3.PyLong_AsLong(sbp)
	dbpI = python3.PyLong_AsLong(dbp)
	python3.PyGILState_Release(gstate)
	return
}

func Calc(dataFilePath string) {
    CalcPre(dataFilePath)
}

Sample caller like this: go Calc("aaa.csv").

Describe what you expected: We can call go Calc() many times without a fatal error.

Steps to reproduce the issue: Use the code above and environments above, put these code into a goroutine, like go Calc("aaa.csv").For simplicity, you may just remove the algorithm part and just have the skeleton remained.

Jun 29 '20 02:06 Zalberth

Every thing is fine when go Calc("aaa.csv") called the first time. But the fatal error emits the second time when go Calc("aaa.csv") is called.

Jun 29 '20 02:06 Zalberth

I haven't looked at your code in detail, but one thing I've noticed is the absence of any <PyObject>.DecRef() calls. Python objects referenced in Go should get decref'ed when they're no longer needed (i.e. bArgs.DecRef()).

Sep 24 '20 01:09 christian-korneck

It seems we could only PyImport_ImportModule some module for only one time. And even after that, calling functions in multiple goroutines still hit similar problems.

Oct 14 '20 07:10 chinajuanbob

Describe what happened: Environments: On MacOS (Catalina Version 10.15.4) Python3.7.6 Go1.13.8 I want to use go-python3 to invoke an algorithm written in Python3, but as described, an fatal error will generated when the second time I invoke this algorithm. From the output message, it seems that PyImport_ImportModule causes this error.

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xa pc=0x91256a3]

runtime stack:
runtime.throw(0x4967a75, 0x2a)
        /usr/local/go/src/runtime/panic.go:774 +0x72
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:378 +0x47c

goroutine 41 [syscall]:
runtime.cgocall(0x4637740, 0xc000063c18, 0x48a4660)
        /usr/local/go/src/runtime/cgocall.go:128 +0x5b fp=0xc000063be8 sp=0xc000063bb0 pc=0x4004d0b
github.com/DataDog/go-python3._Cfunc_PyImport_ImportModule(0x8061d90, 0x0)
        _cgo_gotypes.go:3780 +0x4a fp=0xc000063c18 sp=0xc000063be8 pc=0x462c2fa
github.com/DataDog/go-python3.PyImport_ImportModule(0x49501f5, 0x8, 0x0)
        /Users/zhao/go/pkg/mod/github.com/!data!dog/[email protected]/import.go:24 +0x87 fp=0xc000063c80 sp=0xc000063c18 pc=0x462e267
PPGServer/pkg/algo.ImportModule(0x4964926, 0x26, 0x49501f5, 0x8, 0x1)
        /Users/zhao/go/src/PPGServer/pkg/algo/ppg.go:42 +0x4cb fp=0xc000063d98 sp=0xc000063c80 pc=0x46332db
PPGServer/pkg/algo.CalcPre(0xc0003560c0, 0xd, 0x0, 0x0)
....

Here is the sample code. A wrapper of PyImport_ImportModule:

// ImportModule will import python module from given directory
func ImportModule(dir, name string) *python3.PyObject {
	fmt.Println("python3.PyImport_ImportModule before")
	sysModule := python3.PyImport_ImportModule("sys") // import sys
	fmt.Println("python3.PyImport_ImportModule success")
	path := sysModule.GetAttrString("path")           // path = sys.path
	ob := python3.PyList_GetItem(path, 1)
	fmt.Println("check:", python3.PyUnicode_Check(ob))
	fmt.Println("path:", python3.PyUnicode_AsUTF8(ob))
	fmt.Println("sysModule.GetAttrString success")
	python3.PyList_Insert(path, 0, python3.PyUnicode_FromString("/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages"))
	python3.PyList_Insert(path, 0, python3.PyUnicode_FromString(dir))
	fmt.Println("After module insert:", python3.PyUnicode_AsUTF8(python3.PyList_GetItem(path, 0)))
	fmt.Println("module name:", name)
	return python3.PyImport_ImportModule(name) 
}

Each time the algorithm is called in a goroutine.

func CalcPre(dataFilePath string) (sbpI int, dbpI int) {
	python3.Py_Initialize()
	if !python3.Py_IsInitialized() {
		fmt.Println("Error initializing the python interpreter")
		os.Exit(1)
	}
	gstate = python3.PyGILState_Ensure()
	fmt.Println("Py_Initialize success")
	vbp := ImportModule("/Users/zhao/Desktop/lab/ppython", "value_bp")
	fmt.Println("ImportModule success")

	b := vbp.GetAttrString("estimate")
	fmt.Printf("[FUNC] b = %#v\n", b)
	bArgs := python3.PyTuple_New(1)
	python3.PyTuple_SetItem(bArgs, 0, python3.PyUnicode_FromString(dataFilePath))
	re := b.Call(bArgs, python3.Py_None)
	sbp := python3.PyTuple_GetItem(re, 0)
	dbp := python3.PyTuple_GetItem(re, 1)
	defer func() {
		python3.Py_Finalize()
		fmt.Println("python3.Py_Finalize()")
	}()
	sbpI = python3.PyLong_AsLong(sbp)
	dbpI = python3.PyLong_AsLong(dbp)
	python3.PyGILState_Release(gstate)
	return
}

func Calc(dataFilePath string) {
    CalcPre(dataFilePath)
}

Sample caller like this: go Calc("aaa.csv").

Describe what you expected: We can call go Calc() many times without a fatal error.

Steps to reproduce the issue: Use the code above and environments above, put these code into a goroutine, like go Calc("aaa.csv").For simplicity, you may just remove the algorithm part and just have the skeleton remained.

Hello, do you have a solution to this problem?

Nov 06 '20 10:11 javyxu

maybe as a starting point:

here's an example for creating a Go function around a Python function that you can loop over (= it works a second time): code / explanation
if you add locking and releasing the GIL and locking the thread to it you should be able to use it in a go routine. Here's a example (it uses the sbinet/go-python instead of datadog/go-python3 but you get the idea). code / explanation

Nov 06 '20 10:11 christian-korneck

go-python3 go-python3 copied to clipboard

python3.PyImport_ImportModule(name) will emit a fatal error when called the second time.

go-python3
go-python3 copied to clipboard