go-sqlite3 icon indicating copy to clipboard operation
go-sqlite3 copied to clipboard

Seserialize and Deserialize failed for the size greater than 2G

Open KaitoHH opened this issue 10 months ago • 0 comments

When the size of db exceeds 2G (approximately the size ofint32), the call of Serialize (also Deserialize) will fail.

Here is a minimal poc to reproduce the problem

package main

import (
	"context"
	"database/sql"
	"fmt"
	"os"
	"strings"

	"github.com/mattn/go-sqlite3"
)

func must(err error) {
	if err != nil {
		panic(err)
	}
}

func main() {
	db, err := sql.Open("sqlite3", ":memory:")
	must(err)

	_, err = db.Exec("create table test (id integer primary key, name text)")
	must(err)

	b := strings.Builder{}
	for i := 0; i < 10000; i++ {
		b.WriteString("a")
	}

	rows := 200000
	// rows := 300000 // changed rows to 300000 cause serialize to fail
	for i := 0; i < rows; i++ {
		_, err = db.Exec("insert into test (name) values (?)", b.String())
		must(err)
	}

	// serialize to file
	conn, err := db.Conn(context.Background())
	must(err)

	err = conn.Raw(func(driverConn any) error {
		sqliteConn := driverConn.(*sqlite3.SQLiteConn)
		bs, err := sqliteConn.Serialize("")
		if err != nil {
			return err
		}
		f, err := os.Create("poc.db")
		if err != nil {
			return err
		}
		_, err = f.Write(bs)
		return err
	})
	must(err)

	// deserialize from file
	db, err = sql.Open("sqlite3", ":memory:")
	must(err)
	conn, err = db.Conn(context.Background())
	must(err)
	err = conn.Raw(func(driverConn any) error {
		sqliteConn := driverConn.(*sqlite3.SQLiteConn)
		bs, err := os.ReadFile("poc.db")
		if err != nil {
			return err
		}
		return sqliteConn.Deserialize(bs, "")
	})
	must(err)
	conn.Close()

	r, err := db.Query("select count(*) from test")
	must(err)
	defer r.Close()
	var count int
	r.Next()
	must(r.Scan(&count))
	fmt.Println(conut)

}

I did not look too deep of the issues, but I suspected the problem may related to the conversion between uint64 and uintptr. I noticed the sqlite3_malloc64 in https://github.com/mattn/go-sqlite3/blob/7658c06970ecf5588d8cd930ed1f2de7223f1010/sqlite3_opt_serialize.go#L69 returns null when the length of b is greater than 2G.

KaitoHH avatar Dec 18 '24 09:12 KaitoHH