Data Loader: Creating a Universal Data Loader (API) framework that could be expanded to load all types of Waltz data

[TL41]

Description

We can collaborate on building this out, but we would like to get your thoughts on how the framework/foundation for this should look. Let us know if there's a simple/basic table we could start out on, and go from there

---

Leading Point has been an external (w/r/t Waltz) data loader that can load many of the Waltz Data tables.

• Org
• Persons
• Application
• Measurables
• Physical Flow
• Logical Flow

We can also do the required links between these entities. (i.e. Apps-to-Org, Org-to-Persons, Persons-to-Org, etc)

However, we typically are typically calling this loader from a command line and passing in the params. We would like to move towards universal loading APIs.

There should be a clear line drawn: (1) Institution specific "data conversion" code that would transform data into the preset Waltz load format (2) Waltz Loading API which consume and load the Waltz load format

This way, Waltz doesn't care what your source of data is: spreadsheets, vendor products, internal systems, etc - you convert it, Waltz will load it.

as an example, this is our JSON format for Application table data (some 'application' table columns are not included here)

[
{"asset_code":"111111",
"kind":"CUSTOMISED",
"name":"Application001",
"provenance":"TESTCASE",
"description":"Description for Application001", 
"lifecycle_phase":"UNKNOWN",
"organisational_unit_external_id":"BU_0000",
"business_criticality":"VERY_HIGH" },
{"asset_code":"222222",
"kind":"CUSTOMISED",
"name":"Application002",
"provenance":"TESTCASE",
"description":"Description for Application002", 
"lifecycle_phase":"UNKNOWN",
"organisational_unit_external_id":"BU_0000",
"business_criticality":"VERY_HIGH" }
]

Resourcing

We would like to collaborate on this feature

[davidwatkins73]

Some short thoughts :)

• need to think about optional (or pseudo optional) fields such as business criticality
• standardize naming (not nesc same as underlying tables, i.e. asset_code should be external_id
• external id's should be mandatory (regardless of the underlying model)
• ...