Fix stacked borrows violations

[clubby789]

Use offsets from a single pointer instead of multiple pointers into an array. Also removed an integer-pointer cast which can cause issues with Miri