Proposal for enhancing historical information of Operations in FireFly for DX and other plugins

[gabriel-indik]

When data exchange events are received, they contain additional information under the "info" JSON object. Within a single operation there may be more than one data exchange event. Currently, when a new data exchange event comes in for an operation that already had a previous data exchange event, the "info" JSON object is overwritten. The logic should be changed in order to aggregate these objects as it will be important to see what that information was throughout the operation and in each of the data exchange events. More information can be found in https://github.com/hyperledger/firefly/pull/534#pullrequestreview-886324831

[peterbroadhurst]

Just replaying my understanding.

Current behavior of FireFly operations as used by EVM Connect and FireFly Data Exchange implementations:

• Operations have three states Pending,Succeeded, Failed
• There are output and error fields to capture implementation specific detail related to these states
• Plugin implementations (and by extension APIs exposed by those plugins to other microservices) replace the entire output section, each time there is a status change
• Multiple updates can happen without changing states
• The plugin implementation is responsible for merging any historical information into the latest state
• Only the latest copy is kept in the database

Proposed enhancement:

• Change the database structure for operations to have multiple output/error operation_status change records
• Update the API to query these historical records
• Change connectors like DX/EVMConnect that currently aggregate this data, to just provide minimal update history

Note that in EVMConnect, many of updates might occur for a single operations - as it works through submission, gas estimation, signing+sending to get a TX hash, confirmation tracking, possible re-submission at higher gas prices, obtaining a receipt, and finally getting confirmation.