flatty
flatty copied to clipboard
treeform
Flatty - tools and serializer for plain flat binary files.
Flatty - serializer and tools for flat binary blobs.
atlas use flattynimble install flatty
This library has no dependencies other than the Nim standard library.
About
- Aim of
flattyis to be the fastest and simplest serializer/deserializer for Nim. - Also includes
hexprintto print out binary data. - Also includes
binnya simpler replacement for StringStream (no IO effects, operates on a string) - Also includes
hashya hash for any objects based on the flatty serializer. - Also includes
encodea way to convert to/from utf16 BE/LE and with BOM and utf32.
Speed
Flatty aims to be fast. It achieves this by:
- Not using slowish StringStream.
- Checking if objects are "flat" and just copying them.
- Not doing anything extra like versioning, type checking, etc ...
- Liberal use of
{.inline.}
Serialize speed
name ............................... min time avg time std dv runs
treeform/flatty .................... 6.303 ms 6.514 ms ±0.181 x100
bingod/planetis-m ................. 11.337 ms 12.688 ms ±1.641 x100
disruptek/frosty .................. 14.767 ms 14.924 ms ±0.122 x100
treeform/jsony .................... 12.989 ms 13.343 ms ±0.408 x100
Deserialize speed
treeform/flatty ................... 10.526 ms 16.134 ms ±5.508 x100
bingod/planetis-m binTo ........... 12.836 ms 17.993 ms ±0.181 x100
disruptek/frosty .................. 38.513 ms 42.357 ms ±0.535 x100
treeform/jsony .................... 96.830 ms 100.615 ms ±0.992 x100
JavaScript
Flatty supports Nim's js mode. Some features like uint64/int64 are supported badly because of Nim's limitations. Serializing of non-Nim JavaScript objects is not supported.
Versioning
Note, unlike protobuf or thirft, flatty has no versioning mechanism, if structure of your objects changes the resulting binary would be changed and could not be read back again. Because the schema is just plain Nim types, you need to make sure changing them does not impact your ability to read old flatty binary blobs.
Compression
Flatty does not do compression it self but I recommend using the excellent https://github.com/guzba/supersnappy library to compress your flatty blobs before you send them over network or write them to disk. Snappy protocol is at this sweet spot of very fast compression, not the best but decent compression ratio, and very simple code.
Networking
The flatty + supersnappy + netty were originally made to be used together. Netty is a great for UDP networking for games.
API: flatty
import flatty
func toFlatty
Takes structures and turns them into binary string.
func toFlatty[T](x: T): string
func fromFlatty
Takes binary string and turn into structures.
func fromFlatty[T](s: string; x: typedesc[T]): T
API: flatty/hexprint
import flatty/hexprint
proc hexPrint
Prints a string in hex format of the old DOS debug program. Useful for looking at binary dumps.
hexPrint("Hi how are you doing today?")
0000: 48 69 20 68 6F 77 20 61-72 65 20 79 6F 75 20 64 Hi how are you d
0010: 6F 69 6E 67 20 74 6F 64-61 79 3F .. .. .. .. .. oing today?.....
proc hexPrint(buf: string): string
API: flatty/binny
import flatty/binny
func readUint8
func readUint8(s: string; i: int): uint8
func writeUint8
func writeUint8(s: var string; i: int; v: uint8)
func addUint8
func addUint8(s: var string; v: uint8)
func readUint16
func readUint16(s: string; i: int): uint16
func writeUint16
func writeUint16(s: var string; i: int; v: uint16)
func addUint16
func addUint16(s: var string; v: uint16)
func readUint32
func readUint32(s: string; i: int): uint32
func writeUint32
func writeUint32(s: var string; i: int; v: uint32)
func addUint32
func addUint32(s: var string; v: uint32)
func readUint64
func readUint64(s: string; i: int): uint64
func writeUint64
func writeUint64(s: var string; i: int; v: uint64)
func addUint64
func addUint64(s: var string; v: uint64)
func readInt8
func readInt8(s: string; i: int): int8
func writeInt8
func writeInt8(s: var string; i: int; v: int8)
func addInt8
func addInt8(s: var string; v: int8)
func readInt16
func readInt16(s: string; i: int): int16
func writeInt16
func writeInt16(s: var string; i: int; v: int16)
func addInt16
func addInt16(s: var string; i: int16)
func readInt32
func readInt32(s: string; i: int): int32
func writeInt32
func writeInt32(s: var string; i: int; v: int32)
func addInt32
func addInt32(s: var string; i: int32)
func readInt64
func readInt64(s: string; i: int): int64
func writeInt64
func writeInt64(s: var string; i: int; v: int64)
func addInt64
func addInt64(s: var string; i: int64)
func readFloat32
func readFloat32(s: string; i: int): float32
func addFloat32
func addFloat32(s: var string; v: float32)
func readFloat64
func readFloat64(s: string; i: int): float64
func addFloat64
func addFloat64(s: var string; v: float64)
func addStr
func addStr(s: var string; v: string)
func readStr
func readStr(s: string; i: int; v: int): string
func swap
func swap(v: uint8): uint8
func swap
func swap(v: uint16): uint16
func swap
func swap(v: uint32): uint32
func swap
func swap(v: uint64): uint64
func maybeSwap
func maybeSwap[T](v: T; enable: bool): T
treeform