velocypack

velocypack copied to clipboard

Is it possible to support typed arrays (packed arrays). Each element of an array is of known type and is not preceded by the byte of its type. It will reduce the size of large homogeneous arrays.

It is implemented in cbor, see https://tools.ietf.org/html/draft-ietf-cbor-array-tags-00

There is an optimization in VelocyPack for arrays in which all array members have the same size. This is the array types 0x02 to 0x05 (including). As mentioned, this only works when all array members have the same size, not the same type.

It is not looking for member types, because members with the same type can still have different sizes, e.g. two strings (same type) can have different lengths (different sizes). That said, in VelocyPack two strings with different lengths will also be two different VelocyPack types, at least if the strings are longer than 127 bytes.

The optimization will be pulled off automatically when it is detected that array members are homogenous in size.

Thank you, for your reply. I'm aware of this optimization. It doesn't work well for my use case. I need to pass around large arrays of floats and uint64 (large values, so cant benefit from varint). In case of uint64, I cant always benefit from this optimization, since even if only one value has size 7bytes, than its not possible to use types 0x02 to 0x05. I'd like to specify fixed size for all elements of this array like fixed_array_64.

VelocyPack currently doesn't support this. Note that there are also compact Arrays which use the least possible storage space. Values will be added one after the other without any padding. Iterating over the values in this array should be fast (just a linear scan), but random access to a specific array element will have O(n) complexity.