go-sqlite3
go-sqlite3 copied to clipboard
Seserialize and Deserialize failed for the size greater than 2G
When the size of db exceeds 2G (approximately the size ofint32), the call of Serialize (also Deserialize) will fail.
Here is a minimal poc to reproduce the problem
package main
import (
"context"
"database/sql"
"fmt"
"os"
"strings"
"github.com/mattn/go-sqlite3"
)
func must(err error) {
if err != nil {
panic(err)
}
}
func main() {
db, err := sql.Open("sqlite3", ":memory:")
must(err)
_, err = db.Exec("create table test (id integer primary key, name text)")
must(err)
b := strings.Builder{}
for i := 0; i < 10000; i++ {
b.WriteString("a")
}
rows := 200000
// rows := 300000 // changed rows to 300000 cause serialize to fail
for i := 0; i < rows; i++ {
_, err = db.Exec("insert into test (name) values (?)", b.String())
must(err)
}
// serialize to file
conn, err := db.Conn(context.Background())
must(err)
err = conn.Raw(func(driverConn any) error {
sqliteConn := driverConn.(*sqlite3.SQLiteConn)
bs, err := sqliteConn.Serialize("")
if err != nil {
return err
}
f, err := os.Create("poc.db")
if err != nil {
return err
}
_, err = f.Write(bs)
return err
})
must(err)
// deserialize from file
db, err = sql.Open("sqlite3", ":memory:")
must(err)
conn, err = db.Conn(context.Background())
must(err)
err = conn.Raw(func(driverConn any) error {
sqliteConn := driverConn.(*sqlite3.SQLiteConn)
bs, err := os.ReadFile("poc.db")
if err != nil {
return err
}
return sqliteConn.Deserialize(bs, "")
})
must(err)
conn.Close()
r, err := db.Query("select count(*) from test")
must(err)
defer r.Close()
var count int
r.Next()
must(r.Scan(&count))
fmt.Println(conut)
}
I did not look too deep of the issues, but I suspected the problem may related to the conversion between uint64 and uintptr. I noticed the sqlite3_malloc64 in https://github.com/mattn/go-sqlite3/blob/7658c06970ecf5588d8cd930ed1f2de7223f1010/sqlite3_opt_serialize.go#L69 returns null when the length of b is greater than 2G.