Audit & document `unsafe` usages

[mccolljr]

There are a few usages of unsafe - I tried to cover these cases in tests, but I need to make a pass through the code to add // Safety: ... comments + triple-check that things are doing exactly what I think they are.