delta-rs

delta-rs copied to clipboard

This is likely a low priority issue, but is mentioned in the TODO list and is an aspect of the delta log protocol that is not yet implemented in delta-rs. As described in the protocol doc, commit provenance information, may be included in delta logs within a commitInfo action so delta-rs should be able to read and write these.

The reference implementation creates this action and prepends it to the others before writing the log entry.

A TODO exists in DeltaTransaction where we should be able to do something similar.

Closing the dupe I just created in #152

This would be great to expose to the DeltaTransaction writer code such that a caller can supply their own serde_json::Value (map) of JSON that can be addied into the transaction log.

The use-case I have in mind for this is to enable improved auditability of transactions, for example kafka-delta-ingest could add a map such as:

{
  "commitInfo": {
    "kafka-delta-ingest": {
      "version" : "0.2.3",
      "delta-rs-version": "0.1.3",
      "topic": "originating-kafka-topic",
      "partition": 1
    }
  }
}

Implementing that in #265, @rtyler please link the PR also to this issue.

@rtyler what you're mentioning about kafka ingest is called "userMetadata" and it is a string. We can't get a struct there.

@nfx based on https://github.com/delta-io/delta/blob/master/PROTOCOL.md#commit-provenance-information, commit info action is supposed to be schemaless and can store arbitrary json object right?

@houqp we'd need to check to what extent is it arbitrary, as delta history technically returns commit infos.

We do have the commit info action supported now, clearing this out