clkhash icon indicating copy to clipboard operation
clkhash copied to clipboard

Support binary uploads to entity-service

Open hardbyte opened this issue 5 years ago • 0 comments

An experimental api has been added for uploading CLKs as a binary file. This is to allow for faster and more efficient data transfer. The same rest endpoint (/projects/{project_id}/clks) is used.

Additional headers are required:

Content-Type: application/octet-stream
Hash-Count: XX
Hash-Size: YY

The uploaded stream is simply a packed serialization of (index, CLK data, popcount):

bit_packing_fmt = "!1I<Hash-Size>s1H"

Where:

  • "!" Use network byte order (big-endian).
  • "1I" Store the index in 4 bytes as an unsigned int.
  • <Hash-Size>s the length of a hash in Bytes. Default value is "128s". Store the 128 raw bytes of the bitarray.
  • "1H" The popcount stored in 2 bytes (unsigned short)

https://docs.python.org/3/library/struct.html#format-strings

Aha! Link: https://csiro.aha.io/features/ANONLINK-26

hardbyte avatar Aug 09 '18 23:08 hardbyte