Use unsafe for most of the FFI

[thumphries]

... except the stuff that uses FunPtr.

Important for throughput, but would like to have real tests before doing so