Add a useful and interoperable Data representation to CALM

[ojeb2]

Feature Request

Description of Problem:

CALM doesn’t currently provide enough definitional detail for Data in solution designs. We should be able to integrate CALM designs into popular data catalog definitions as well as represent data dependencies across multiple systems and services.

Potential Solutions:

There are a number of standards data definitions, including DPROD, CDMC and FinOS CCC data services that should be considered. In the Oct 7th hack day we considered extending Node type with CDMC DataAsset or DPROD Data Product.

[arhy]

Some additional content here from @ojeb2 on https://github.com/finos/architecture-as-code/issues/450#issuecomment-2403346305.

Propose we remove database from node type, introduce data-asset. A database is then a node of type service, with data-assets deployed-in.

Similarly, a reference data service could contain multiple data-assets.

Keeping data-asset as a node will mean controls can be modelled on top.

[arhy]

For example - here is what I think traderx might look like. Have used composed-of vs deployed-in.

{
  "nodes": [
    {
      "unique-id": "traderx-database",
      "node-type": "service",
      "name": "TraderX Database",
      "description": "Database for storing account, user, position, and trade information."
    },
    {
      "unique-id": "accounts-table",
      "node-type": "data-asset",
      "name": "Accounts Table",
      "description": "Table for storing account information.",
      "detailed-architecture": "CREATE TABLE Accounts ( ID INTEGER PRIMARY KEY, DisplayName VARCHAR (50) );"
    },
    {
      "unique-id": "account-users-table",
      "node-type": "data-asset",
      "name": "AccountUsers Table",
      "description": "Table for storing account user information.",
      "detailed-architecture": "CREATE TABLE AccountUsers ( AccountID INTEGER NOT NULL, Username VARCHAR(15) NOT NULL, PRIMARY KEY (AccountID, Username)); ALTER TABLE AccountUsers ADD FOREIGN KEY (AccountID) References Accounts(ID);"
    },
    {
      "unique-id": "positions-table",
      "node-type": "data-asset",
      "name": "Positions Table",
      "description": "Table for storing position information.",
      "detailed-architecture": "CREATE TABLE Positions ( AccountID INTEGER , Security VARCHAR(15) , Updated TIMESTAMP, Quantity INTEGER, Primary Key (AccountID, Security) ); ALTER TABLE Positions ADD FOREIGN KEY (AccountID) References Accounts(ID);"
    },
    {
      "unique-id": "trades-table",
      "node-type": "data-asset",
      "name": "Trades Table",
      "description": "Table for storing trade information.",
      "detailed-architecture": "CREATE TABLE Trades ( ID Varchar (50) Primary Key, AccountID INTEGER, Created TIMESTAMP, Updated TIMESTAMP, Security VARCHAR (15) , Side VARCHAR(10) check (Side in ('Buy','Sell')), Quantity INTEGER check Quantity > 0 , State VARCHAR(20) check (State in ('New', 'Processing', 'Settled', 'Cancelled'))); ALTER TABLE Trades ADD FOREIGN KEY (AccountID) references Accounts(ID);"
    }
  ],
  "relationships": [
    {
      "unique-id": "traderx-database-tables",
      "relationship-type": {
        "composed-of": {
          "container": "traderx-database",
          "nodes": [
            "accounts-table",
            "account-users-table",
            "positions-table",
            "trades-table"
          ]
        }
      }
    }
  ]
}

[LeighFinegold]

@arhy, @ojeb2 - Any chance you can play this further forward in relation to DataService interface? Even taking the Load Account use case on https://github.com/finos/traderX/blob/main/docs/flows.md would aid my understanding on how you plan to combine the data-asset above with how the flow of account information goes from the account service component to the web client.

[jpgough-ms]

@arhy the intention of detailed-architecture is to link to another CALM architecture document (rather than contain SQL).

A way of modelling what you are doing here would be to use an interface that has a mandatory field for SQL statement.

[arhy]

@LeighFinegold yes

@jpgough-ms cool - will rework in a real example.

[rocketstack-matt]

Closing in favor of https://github.com/finos/architecture-as-code/issues/1371