enrich icon indicating copy to clipboard operation
enrich copied to clipboard

Common: explore graph extensions to Avro enriched event

Open chuwy opened this issue 4 years ago • 1 comments

We may want to leap straight to this approach. The basic idea would be to express relationships between entities in the event as typed edges.

Need to review Nathan Marz's chapter on modeling graphs in Thrift in Big Data.

Here is a simple example (using self-describing JSON instead of Avro for simplicity):

{
  "nodes": {
   "n0": {
      "schema": "iglu:com.acme/player/jsonschema/1-0-0",
      "data": {
        "username": "bob",
        "alliance": "pirates"
      }
    },
    "n1": {
      "schema": "iglu:com.acme/player/jsonschema/1-0-0",
      "data": {
        "username": "sue",
        "alliance": "zombies"
      }
    },
    "n2": {
      "schema": "iglu:com.acme/kills/jsonschema/1-0-0",
      "data": {}
    },
    "n3": {
      "schema": "iglu:com.acme/level/jsonschema/1-0-0",
      "data": {
        "name": "blood gulch",
        "difficulty": 23
      }
    },
  },
  "edges": {
    "e0": {
      "schema": "iglu:com.snowplowanalytics.snowplow/verb-subject/jsonschema/1-0-0",
      "source": "n0",
      "target": "n1"
    },
   "e1":  {
      "schema": "iglu:com.snowplowanalytics.snowplow/verb-direct-object/jsonschema/1-0-0",
      "source": "n1",
      "target": "n2"
    },
   "e2":  {
      "schema": "iglu:com.acme/at-geographic/jsonschema/1-0-0",
      "source": "n2",
      "target": "n3"
    }
  }
}

chuwy avatar Jun 19 '20 16:06 chuwy