Scanning a Vector from SQLite Back to Go

[wjkoh]

I store []float32 data using sqlite_vec's SerializeFloat32 function and am wondering how I can retrieve it from SQLite into Go. I've been using vec_to_json and unmarshaling the JSON on the Go side as the following code, but I suspect there might be a better approach. Could you please advise?

	var embeddingJson string
	row := db.QueryRowContext(
		ctx,
		`SELECT vec_to_json(embedding) FROM documents WHERE document_id = ?`,
		id,
	)
	err := row.Scan(&embeddingJson)
	if err != nil {
		return nil, err
	}
	var embedding []float32
	err = json.Unmarshal([]byte(embeddingJson), &embedding)
	if err != nil {
		return nil, err
	}
	return embedding, nil

[asg017]

@wjkoh does something like this work?

func DeserializeFloat32(data []byte) ([]float32, error) {
    if len(data)%4 != 0 {
        return nil, fmt.Errorf("invalid data length: must be a multiple of 4")
    }

    // Create a slice to hold the deserialized float32 values
    count := len(data) / 4
    vector := make([]float32, count)

    buf := bytes.NewReader(data)
    err := binary.Read(buf, binary.LittleEndian, &vector)
    if err != nil {
        return nil, err
    }

    return vector, nil
}

This compliments the SerializeFloat32() method, but I haven't tested it yet.

If it works, I'll add it to the Go library bindings

[wjkoh]

Thanks for your reply, @asg017! I think that your function and https://pkg.go.dev/database/sql#RawBytes can work together for this use case. I'll give it a try soon!

[wjkoh]

@asg017 Thank you for the function! It works as intended and is very helpful for unit testing and debugging. I hope you add the function to the next release.