ESON, Exa-scale Storage Object Notation

ESON is simple but powerful schema-less binary data format designed to handle Exa-scale data. Example includes graphics(e.g. geometry, volume and textures) and may also applicable for in-memory database and scientific data.

ESON is also designed to handle large-scale data efficiently for comming NVM(non-volatile memory) or SCM(storage-class memory) era.

C++ and python API is primarily provided.

Version

• 0.3.2 (Nov 2015) C++ API is now header only!
• 0.3.1 (Aug 2015) Initial python2 and python3 binding(no native module compilation required)
• 0.3.0 (Mar 2015) Initial support of ARRAY in C/C++ API
• 0.2.0 (Jan 2015)
• 0.1.0 (Jul 2013)

Status

Very unstable. Spec and API will change in the future.

Quick tutorial

$ make
$ ./eson_test

Specification

See SPECIFICATION.md

Design and implementation references

ESON design is strongly affected by

• BSON http://bsonspec.org/

ESON C++ API is strongly affected by

• picojson https://github.com/kazuho/picojson

Example in C++

// Define this only in *one* .cc
#define ESON_IMPLEMENTATION
#include "eson.h"

#include <iostream>
#include <cstdlib>
#include <cstdio>

static void
ESONTest()
{
  eson::Value v;
  double dbl = 1.234;
  eson::Value vd(dbl);

  double dbl2 = 3.4;
  eson::Value vd2(dbl2);

  int64_t i = 144;
  eson::Value ival(i);

  std::string name("jojo");
  eson::Value sval(name);

  char bindata[12];
  for (int i = 0; i < 12; i++) {
    bindata[i] = i;
  }
  eson::Value bval((const uint8_t*)bindata, 12);

  eson::Object o;
  o["abora"] = vd;
  o["muda"] = vd2;
  o["dora"] = ival;
  o["name"] = sval;
  o["bin"] = bval;

  v = eson::Value(o);

  // First calcuate required size for serialized data.
  int64_t sz = v.Size();

  uint8_t* buf = new uint8_t[sz]; // or use mmap() if sz is large.
  uint8_t* ptr = &buf[0];

  ptr = v.Serialize(ptr);
  assert((ptr-&buf[0]) == sz);

  FILE* fp = fopen("output.eson", "wb");
  fwrite(buf, 1, sz, fp);
  fclose(fp);

  eson::Value ret;
  std::string err = eson::Parse(ret, buf);
  if (!err.empty()) {
    std::cout << "err:" << err << std::endl;
  }

  eson::Value dval = ret.Get("muda");
  printf("muda = %f\n", dval.Get<double>());

  eson::Binary bin = ret.Get("bin").Get<eson::Binary>();
  printf("bin len = %d\n", bin.size);
  for (int i = 0; i < bin.size; i++) {
    printf("    bin[%d] = %d\n", i, bin.ptr[i]);
  }

  delete buf;
}

Example in JavaScript(node.js)

var eson = require('eson-binary');
var fs = require('fs');

if (process.argv.length < 3) {
  console.log("needs input.eson");
  process.exit(-1);
}

var buf = fs.readFileSync(process.argv[2])
var b = eson.parse(buf);

console.log(b)

Note

How to handle int16, int32, fp16, float32 data?

Please use BINARY data type.

TODO

• [ ] Support Endianness.
• [ ] Efficiently serialize key table for better search performance.
• [ ] Make API Zero-Copy to reduce memory.
• [ ] Add serialize API in JavaScript API.
• [ ] Support 2GB+ size in JavaScript API.

Compression

Currently we are planning to use zstd or LZ4 compression for lossless binary data. Lossy compression for floating point data is interesting direction to explore. There are zfp an fpzip for lossy floating point compression.

https://computation.llnl.gov/projects/floating-point-compression

Author(s)

• Syoyo Fujita([email protected])
• Yasutoshi Mori(https://github.com/mirageym) Python binding.

License

ESON C++ library and JavaScript library is licensed under MIT license. ESON python binding is licensed under 3-clause BSD.

Third-party licenses

• BSON Python binding. 3-clause BSD.
• lz4 is licensed under 2-clause BSD license.