Does Gorm crash on multiple concurrent writes, or is it my code ?

[adnseek]

Your Question

i have written a Golang Import Script that uses a workerpool for concurrent usage, together with the great GORM as ORM Layer.

But everytime i run my code with more than 2 workers, at some point Gorm crashes with a segmentation fault.

Expected answer

Now i want to know if it is definitely my code ? or does anyone of you experience the same ?

Cheers

Adrian

[adnseek]

panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1069701]

goroutine 48319 [running]: github.com/go-sql-driver/mysql.(*mysqlStmt).writeExecutePacket(0xc0005a6018, {0xc00020c480, 0x24, 0x104d152?}) /Users/adriangier/go/pkg/mod/github.com/go-sql-driver/[email protected]/packets.go:1102 +0x1d33 github.com/go-sql-driver/mysql.(*mysqlStmt).Exec(0xc0005a6018, {0xc00020c480?, 0xc000028080?, 0x15520c8?}) /Users/adriangier/go/pkg/mod/github.com/go-sql-driver/[email protected]/statement.go:58 +0xb0 github.com/go-sql-driver/mysql.(*mysqlStmt).ExecContext(0xc0005a6018, {0x1551e98, 0xc000028080}, {0xc00023a600, 0x24, 0xc0005a6018?}) /Users/adriangier/go/pkg/mod/github.com/go-sql-driver/[email protected]/connection.go:578 +0x165 database/sql.ctxDriverStmtExec({0x1551e98, 0xc000028080}, {0x15520c8, 0xc0005a6018}, {0xc00023a600?, 0x24, 0x24}) /usr/local/go/src/database/sql/ctxutil.go:65 +0xcc database/sql.resultFromStatement({0x1551e98, 0xc000028080}, {0x1551860, 0xc0004aa120}, 0xc000972df8, {0xc0005b0000, 0x24, 0x40}) /usr/local/go/src/database/sql/sql.go:2663 +0x145 database/sql.(*DB).execDC(0x0?, {0x1551e98, 0xc000028080}, 0xc0000f4090, 0x0?, {0xc0005aa000, 0x273}, {0xc0005b0000, 0x24, 0x40}) /usr/local/go/src/database/sql/sql.go:1700 +0x473 database/sql.(*Tx).ExecContext(0xc0015d0000, {0x1551e98, 0xc000028080}, {0xc0005aa000, 0x273}, {0xc0005b0000, 0x24, 0x40}) /usr/local/go/src/database/sql/sql.go:2491 +0xb7 gorm.io/gorm/callbacks.Create.func1(0xc00143bb90) /Users/adriangier/go/pkg/mod/gorm.io/[email protected]/callbacks/create.go:96 +0x763 gorm.io/gorm.(*processor).Execute(0xc0003f9cc0, 0x6?) /Users/adriangier/go/pkg/mod/gorm.io/[email protected]/callbacks.go:130 +0x433 gorm.io/gorm.(*DB).Create(0xc0009739c0?, {0x13fb700?, 0xc0001e2b40}) /Users/adriangier/go/pkg/mod/gorm.io/[email protected]/finisher_api.go:24 +0xa5 main.importProduct({0xc0015c1380, 0x1a, 0x1a?}, {0x9, 0x14, 0x2, 0xd, 0x3, 0x19, 0xc, ...}, ...) /Users/adriangier/Sites/pipeline_import/main.go:554 +0x1230 main.importProducts.func1() /Users/adriangier/Sites/pipeline_import/main.go:365 +0x85 github.com/gammazero/workerpool.worker(0xc0014b8fd0?, 0x13032c6?, 0xc0009bc580?) /Users/adriangier/go/pkg/mod/github.com/gammazero/[email protected]/workerpool.go:237 +0x2a created by github.com/gammazero/workerpool.(*WorkerPool).dispatch /Users/adriangier/go/pkg/mod/github.com/gammazero/[email protected]/workerpool.go:197 +0x2b8

[jinzhu]

no，are you using one transaction for concurrent writes

[adnseek]

Yes. I just use db.CreateIs this maybe my fault ?Von unterwegs gesendet. Am 24.01.2023 um 02:01 schrieb Jinzhu @.***>: no，are you using one transaction for concurrent writes

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

[adnseek]

Shall I follow this example ?

https://gorm.io/docs/transactions.html

Create a transaction per worker ?

Am 24.01.2023 um 06:57 schrieb Adrian @.***>:

Yes. I just use db.Create

Is this maybe my fault ?

Von unterwegs gesendet.

Am 24.01.2023 um 02:01 schrieb Jinzhu @.***>:

 no，are you using one transaction for concurrent writes — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[adnseek]

I tried to wrap the import in a transaction

At the end of the import (where db is the transaction I hand over with tx

I create the product

But I still receive

After some time.

Any help would be awesome

Thank You Jinzhu

Adrian

Am 24.01.2023 um 09:14 schrieb Adrian Gier @.***>:

Shall I follow this example ?

https://gorm.io/docs/transactions.html

Create a transaction per worker ?

Am 24.01.2023 um 06:57 schrieb Adrian @.***>:

Yes. I just use db.Create

Is this maybe my fault ?

Von unterwegs gesendet.

Am 24.01.2023 um 02:01 schrieb Jinzhu @.***>:

 no，are you using one transaction for concurrent writes — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[adnseek]

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 360 days with no activity. Remove stale label or comment or this will be closed in 180 days