[question] sqlite extension alignment problem in exqlite, but not in sqlite3

[ruslandoga]

👋

I understand that it's unlikely to be a problem in exqlite, but just in case you have any idea as to what I'm doing wrong.

---

Repo: https://github.com/ruslandoga/sqlite-zig-problem

I'm trying to write an extension for sqlite in zig and I'm getting an alignment panic when fetching sqlite's aggregate context. But, I'm only getting it when executing the query from elixir, not from sqlite3 cli.

Meaning,

sqlite> .load dist/extc
sqlite> .load dist/extzig

-- this uses c extension
sqlite> with recursive ten(x) as (select 1 union all select x+1 from ten where x<10) select sumc(x) from ten;
55.0

-- this uses zig extension
sqlite> with recursive ten(x) as (select 1 union all select x+1 from ten where x<10) select sumzig(x) from ten;
55.0

but in elixir:

iex(1)> E.sumc
# [debug] QUERY OK db=0.6ms decode=1.3ms idle=738.5ms
# with recursive ten(x) as (select 1 union all select x+1 from ten where x<10) select sumc(x) from ten []
%Exqlite.Result{
  columns: ["sumc(x)"],
  command: :execute,
  num_rows: 1,
  rows: [[55.0]]
}

iex(2)> E.sumzig
thread 8760720 panic: incorrect alignment

[warmwaffles]

This could actually be a legitimate issue. Can you provide an example Zig file and instructions on how to compile it so I can toy around with it as well?

[warmwaffles]

Oh duh, I should look at the repo you made.

[ruslandoga]

@warmwaffles here's the zig file: https://github.com/ruslandoga/sqlite-zig-problem/blob/master/src/ext.zig

Here's how I'm compiling it on mac: https://github.com/ruslandoga/sqlite-zig-problem/blob/1d0eeecfd50aec8622c70b2c3b928aecb9defc0b/Makefile#L10-L12

Zig docs suggest downloading "nightly" master builds so I'm using

> zig version
0.10.0-dev.2473+e498fb155

[warmwaffles]

@ruslandoga are you also on an M1 mac?

[ruslandoga]

@warmwaffles ah, yes.

[ruslandoga]

@warmwaffles small update: I printed the pointer and it was 102a7acdc when called from Elixir and 120016530 when called from sqlite3-cli. Converting these to base10 yields 4339510492 and 4831929648. And rem(4831929648, 8) = 0 but rem(4339510492, 8) = 4 and that's what (I think) @alignCast checks and panics. Could it be an enif_alloc issue?

-- I switched to c_longlong from f64 for easier debugging and printed the 8 bytes that are supposed to be the aggregate context
sqlite> with recursive ten(x) as (select 1 union all select x+1000 from ten where x<10000) select sumzig(x) from ten;
-- ?*anyopaque, anyopaque@120016530 -- aligned
-- deref: 0 0 0 0 0 0 0 0
-- deref: 1 0 0 0 0 0 0 0
-- ...
-- deref: 227 214 0 0 0 0 0 0
55011

iex(1)> E.sumzig
# ?*anyopaque, anyopaque@106dd2cdc -- not aligned
# deref: 0 0 0 0 0 0 0 0
thread 13267857 panic: incorrect alignment
/Users/q/Developer/copycat/e/src/ext.zig:22:62: 0x106e7274b in getTotal (ext)
  const totalCtxAligned = @alignCast(@alignOf(c_longlong), totalCtx);
                                   ^ src/ext.zig:36:27: 0x106e719bb in sumStep (ext)
  const total = getTotal(ctx) catch return;
                       ^ Panicked during a panic. Aborting.

I was suggested to use std.mem.alignForward, so that's what I'll try next.

---

It doesn't seem to be an m1 issue as the problem is reproducible on github ci's ubuntu-latest.

[ruslandoga]

std.mem.alignForward workaround works, but it'd be easier to not have to use that.

[warmwaffles]

I 100% trust enif_alloc to do the right alignment based on the architecture OTP was compiled for.

[warmwaffles]

@ruslandoga are you still having alignment issues?

[ruslandoga]

@warmwaffles hi! I wasn't able to find the cause of this issue at the time so I moved away from using extensions in my app. I still plan to learn more about low-level programming and find out what's going on, though :)

[warmwaffles]

Alright

[ruslandoga]

👋 @warmwaffles

I think it was due to shifting the address by four bytes (= sizeof(int))

https://github.com/elixir-sqlite/exqlite/blob/baa0924803d41a840b2ac69b2094e2abaae5d51e/c_src/sqlite3_nif.c#L40-L44

Default allocators have pointers like 1080164b0, 8-byte aligned, on a 64bit system. And the custom one in exqlite produces pointers like 1425cb21c, 4-bytes aligned (which is the 8-bytes aligned pointer returned from enif_alloc + 4 bytes to store mem size). I wonder if it can lead to problems where the SQLite tries to allocate and then free more than 4 gigs of memory :)

I also wonder if this slight misalignment affects "performance" in any way, since Zig probably doesn't complain about it out of pure pedantry: https://stackoverflow.com/questions/12491578/whats-the-actual-effect-of-successful-unaligned-accesses-on-x86

[warmwaffles]

Hmmm, I wonder if the ssize_t type would be a better type to use since on a 64 bit system it should be up to 64 bits and on a 32 bit system it should be up to 32 bits.