Research: Arvo format

[rufuspollock]

http://avro.apache.org/docs/current/

New data format. Supported by e.g Google Bigquery (in beta). Some similarities with TDP e.g. JSON schema format. Overall not that similar: focused on optimizations especially designed for HDFS (e.g. block size).

Avro provides:

• Rich data structures.
• A compact, fast, binary data format.
• A container file, to store persistent data.

...

Schema

Avro relies on schemas. When Avro data is read, the schema used when writing it is always present. This permits each datum to be written with no per-value overheads, making serialization both fast and small. This also facilitates use with dynamic, scripting languages, since data, together with its schema, is fully self-describing.

When Avro data is stored in a file, its schema is stored with it, so that files may be processed later by any program. If the program reading the data expects a different schema this can be easily resolved, since both schemas are present.

When Avro is used in RPC, the client and server exchange schemas in the connection handshake. (This can be optimized so that, for most calls, no schemas are actually transmitted.) Since both client and server both have the other's full schema, correspondence between same named fields, missing fields, extra fields, etc. can all be easily resolved.

Avro schemas are defined with JSON . This facilitates implementation in languages that already have JSON libraries.

Types

https://avro.apache.org/docs/1.8.1/trevni/spec.html

null, requires zero bytes. Sometimes used in array columns.
boolean, one bit, packed into bytes, little-endian;
int, like long, but restricted to 32-bit signed values
long 64-bit signed values, represented as above
fixed32 32-bit values stored as four bytes, little-endian.
fixed64 64-bit values stored as eight bytes, little-endian.
float 32-bit IEEE floating point value, little-endian
double 64-bit IEEE floating point value, little-endian
string as above
bytes as above, may be used to encapsulate more complex objects

[kagesenshi]

hi, just stumbled upon frictionlessdata .. was thinking of similar idea to datapackage.json for a while now for a pet project for data exchange platform..

it would be good if datapackage can support avro as an option for resource, as avro schema's schema evolution helps with keeping long term data where the schema might change over time. Avro also is less error prone compared CSV when dealing with dirty data. Additionally, avro schema also support complex nested map/array types.

thanks

[rufuspollock]

@kagesenshi thanks for the feedback and we'll definitely keep this in mind. Note data package already supports avro resources -- it is just a question of how it would integrate with table schema.

[timrobertson100]

+1 for Frictionless providing guidance on the table schemas (both inline or pointing to an avsc file by URL may have benefit?)

(nit: having the typo in this issue title "Arvo" fixed would aid discoverability of it)