Flatty isn't portable relative to endianness

[auxym]

So, I admit this errs on the side of yak shaving, and maintainers should feel free to close the issue if they choose not to support this (maybe with a small warning in the readme).

However, using portable constructs might have very little performance cost. As shown here: https://justine.lol/endian.html, writing "mask and shift" code to read or write integers byte-by-byte gets optimized away by gcc to a single byte swap instruction when needed, or I assume, nothing if no swapping is needing (to be confirmed in godbolt, possibly).

As for offending code, all the to/from flatty procs for integers use direct memory copy. eg: https://github.com/treeform/flatty/blob/master/src/flatty/binny.nim#L15. This means that serialising on a LE cpu, then writing to disk (or a socket, etc), and deserialising on a BE cpu, would presumably result in incorrect data.

And finally for the yak shaving: big endian is almost completely dead. From my research, the following cpu/os combinations might be using big-endian mode:

• ARM and RISC-V are bi-endian, but from my research, no current implementation uses big-endian mode.
• PowerPC running under AIX (supported by Nim in theory) is big-endian.
• SPARC running under Solaris or Linux (supported by Nim in theory) is big-endian.
• Embedded MIPS cpus, somewhat common in network appliances like routers and switches, are mostly big endian.

This might also make testing the issue and resolution somewhat difficult (I wasn't actually able to confirm it, I'm not in possession of any of this hardware, not sure if setting up a VM under eg QEMU would be feasible).

[treeform]

Yes you right flatty does not handle endianness that well. And yes big-endian is super rare and not worth worrying about.

Its done so for speed see here: https://github.com/treeform/flatty/blob/master/src/flatty.nim#L144

An entire seq can be copied in one go even if it has 1000s of integers or any other complex object.

If I had to worry about big endian all of those optimization would not work.

I could just check for big endian and say raise "not supported".

[auxym]

Understood.

A compile-time check and error would be ideal, but I'm not sure it's possible to check endianness from the VM. Alternatively, maybe a single runtime check on flatty import?