Making cgo-based callbacks?

[gwd]

I've got a callback function which, in one of my key queries, is called thousands of times. Initially I did the "easy" thing of making a Golang-based callback, using conn.RegisterFunc(). Profiling revealed that queries were spending the vast majority of their time in the marshalling code.

I recently did some benchmark testing, and found that I can get a 20x speedup in this function by writing it in C, and loading it as an extension. (Benchmarking fingered the golang-based callback as taking nearly 1600ns, and the C-based callback as taking only 80ns.)

However, actually making a separate dynamically-loadable executable is a bit inconvenient for a Go project; I'd have to switch from the built-in Golang build system to a makefile-based system or something. That's a pain, particularly when I just need to define 1 C function and call 1 C function.

What I'd really like to be able to do is something like this:

/*
[Define 'myFunc']
*/
import "C"

/* ... */

func myFuncHook(conn *sqlite3.SQLiteConn) error {
   db := conn.GetContext() // Return underlying SQLiteContext
  ret = C.sqlite3_create_function(db, "myFunc", 3, SQLITE_UTF8, NULL, C.myFunc /* ... */)
  /* ... */
}

And then set myFuncHook as the ConnectionHooks. I see that the underlying C SQLite connection has its own type, but I don't see how to get the connection from the hook callback.

Is that possible? And if it's not possible, could it be made possible?

Alternately, would it make sense to add a RegisterCFunc method, which would take a C function pointer?

I might be able to send you a PR if I knew what kind of solution you'd prefer.

Thanks in advance!

[gwd]

(Alternately, if you think the Golang callback could be sped up significantly, I could post the function and my simple benchmarking code; but my impression was that there was no way to retain the type safety properties without all the overhead of reflection in the unmarshal / marshal paths.)

[rittneje]

https://golang.org/cmd/cgo/

Cgo translates C types into equivalent unexported Go types. Because the translations are unexported, a Go package should not expose C types in its exported API: a C type used in one Go package is different from the same C type used in another.

I don't think we can add something like GetContext() *C.sqlite3 as you proposed. However, I believe something like this could work:

func (c *SQLiteConn) Raw(f func(unsafe.Pointer) error) error {
    c.mu.Lock()
    defer c.mu.Unlock()
    if err := f(unsafe.Pointer(c.db)); err != nil {
        if _, ok := err.(ErrNo); ok {
            return c.lastError()
        }
        return err
    }
    return nil
}

Then your code could do something like:

//#include <sqlite3.h>
import "C"

...

d := &sqlite3.SQLiteDriver{
    ConnectHook: func(c *SQLiteConn) error {
        return c.Raw(func(raw unsafe.Pointer) error {
            rawConn := (*C.sqlite3)(raw)
            if rv := C.sqlite3_create_function(db, ...); rv != C.SQLITE_OK {
                return sqlite3.ErrNo(rv)
            }
            return nil
        }
    },
}

Note that because your code will also need to #include <sqlite3.h>, you will likely want to build with the libsqlite3 build tag (to ensure both your code and this library are looking at the same header file). So we could make it so that the Raw method is only exposed/supported when using that build flag. See https://github.com/mattn/go-sqlite3/blob/master/sqlite3_libsqlite3.go

[gwd]

Thanks for the quick response!

The existence of sqlite3ext.h seems to imply that they're actually working to keep ABI compatibility, and not just API compatibility across versions; so it seems like the worst you might come across is missing symbols at compile time, if you have a newer header that contains functions not present in your older version of the library. So I'm not sure you'd need the libsqlite3 flag restriction.

Anyway, let me play around with it and see what I can come up with.

[gwd]

Well, that was easy. Adding:

func (c *SQLiteConn) Raw(cb func(raw unsafe.Pointer) error) error {
	return cb(unsafe.Pointer(c.db))
}

Seems to do the trick. The hardest part was that due to https://github.com/golang/go/issues/19835, you can't pass a C function pointer back into a C function. So the code above won't work as-is; you have to pass the unwrapped db to a C helper, and have it call sqlite3_create_function.

Let me put this through its paces a bit, and then send a PR.

[gwd]

@rittneje Just making sure I understand your error-handling code. The idea is that the callback can return their own error type, and that's fine, it will be passed through; but if they return a go-sqlite3 error type, then we'll basically take over the error and do the decoding of the extended sqlite3 error codes and so on. Is that right?

[rittneje]

Essentially yes. I imagine that 99% of the time the callback is just using some SQLite library functions, so you really just want to return the error code. But it allows for any arbitrary error for maximum flexibility.