waltz
waltz copied to clipboard
finos
Survey Workflow: design
- [ ] setup ddl
- [ ] migrate current entity_workflow_definition' to 'workflow_definition' (adding external id and owner, status etc...)
- [ ] introduce 'workflow_state' table and populate with the distinct list of state values form the existing 'entity_workflow_state' table
- [ ] migrate 'entity_workflow_state' to 'workflow_instance' (state -> state_id etc)
- [ ] migrate 'entity_workflow_transition' to 'workflow_transition' (drop entity columns, use instance_id and state_id).
- [ ] update existing internal jobs to use the new tables
- [ ] Update / recreate model for new workflow tables
- [ ] Workflow job
- [ ] Externalise rules
- [ ] #5812
- [x] Define predicates for rules
- [ ] Define initialisation rules / predicates
- [ ] Define 'side-effects' for rules e.g. issue surveys or update assessment
- [ ] Workflow admin screens
Users would like the ability to use a workflow system to drive the issuance of surveys with a view to updating one or more assessments in waltz.
For example, an architectural review process: This workflow is used to track architectural governance compliance of change initiatives.
2 assessments will be used:
- Outcome assessment
- Workflow status assessment
An external batch job determines which change initiatives are in scope and sets the outcome assessment to 'In Scope'. A workflow job will wait for assessments in this state and issue a "scoping" survey and update the workflow status assessment to be 'Pending Completion'. etc.... Once the survey is approved the initial assessment is updated to reflect the end state.
Workflow Support
Over the past year we have been using Waltz to support custom workflows. These workflows are orchestrated using bespoke jobs and are not easily reusable. We believe a small set of reusable, composable functionality could simplify the creation future workflows.
An Example workflow
Examples of these bespoke workflow processes have been in areas such as Records Management. The Records Management process coordinates multiple surveys with the goal of providing assessment and rating values against the subject application.
Scoping Survey
The process starts with a job issuing a Scoping survey to determine what, if any, record classes a subject application manages. The scoping survey is issued to the application owner and is reviewed by a representative for the Records Management team. Once approved the record classes are used to determine the next step:
-
No Records:
- a 'does not contain records' assessment is made against the application and the process ends
-
Some Records:
- a 'contains records' assessment is made against the application
for each declared record class a provisional rating is stored as a Records Management Taxonomy Rating for that application
Detail Survey
A secondary job periodically checks the Records Management Taxonomy Rating looking for provisional ratings. For each rating found a new Detailed Survey is issued against the application with a similar set of owners and approvers. As each of the Detail Surveys are approved then the taxonomy ratings are either removed, or the rating moves from provisional to Confirmed.
Observations
The two jobs which issue the Scoping and Detail surveys as intentionally decoupled. They use a combination of Assessments and Measurable (Taxonomy) Ratings to have small, controlled amount of shared state.
Due to this separation, Records Management Administrators can override values in the ratings or assessments and the workflow will act correctly.
For example, an administrator could manually store some provisional record ratings against an application and the corresponding detail surveys would be issued without the need for a scoping survey. Similarly, if an administrator want to manually set some confirmed ratings against an application any outstanding detail survey will be withdrawn.
Towards configurable workflows
If we take the above example and also consider some other use-case we can start cataloguing a set of functionality needed to deliver a flexible workflow solution.
- State management
- Use of assessments and/or taxonomy ratings to track workflow state
- Triggers - when do we need to revaluate the workflow and potentially take action
- submission - when a survey is submittedm but not yet approved
- appproval - when a survey has been approved
- rejection - when a survey has been rejected by the approver
- withdrawn - when a survey has been withdrawn by the owner or approver
- timed - e.g. when a workflow has been in a given state for a set period of time
- Predicates - evaluations to determine which action to take
- Waltz contextual - information about the subject (i.e. investment state, other assessments)
- survey responses - answers given in response to survey questions
- external contextual - information obtained from non-Waltz sources (unlikely to be in phase 1)
- Actions - changes to effect outside of the workflow engine
- update assessment/s
- update rating/s
- update approvers - use case is to allow differing reviewers based on survey answers (i.e. route to the most qualified reviewers)
- issue additional surveys
- add/remove from groups
- external actions - probably simlpy write a basic message to persistent queue for an external job to process
There are the following tables in Waltz which could be modified to accommodate these changes:
- entity_workflow_definition
- entity_workflow_state
- entity_workflow_transition
Most likely by creating a new table to track the state, and renaming entity_workflow_state to entity_workflow_instance
Proposed tables:
workflow_definition
| Attribute | Type | Description |
|---|---|---|
| id | Long | Sequential identifier |
| name | String | name |
| description | String | description |
| external_id | String | external id |
| created_at | Timestamp | workflow first created |
| created_by | String | user to create workflow |
| owning_role | String | which users can modify this workflow |
| status | String | ACTIVE / DRAFT / REMOVED |
workflow_state
| Attribute | Type | Description |
|---|---|---|
| id | Long | identifier |
| workflow_defn_id | Long | FK to workflow_definition |
| name | String | name |
| description | String | description |
| external_id | String | Description |
| state_kind | String | PARKED (Did not fulfil any starting predicate) / IN_PROGRESS /COMPLETED |
workflow_instance
| Attribute | Type | Description |
|---|---|---|
| id | Long | Sequential identifier |
| workflow_definition_id | Long | FK to workflow definition |
| state_id | Long | state of this instance |
| parent_entity_id | Long | Entity this workflow is acting on |
| parent_entity_kind | String | Entity this workflow is acting on |
| external_id | String | external id |
| created_at | Timestamp | workflow first created |
| created_by | String | user to create workflow |
| last_updated_at | Timestamp | when was this instance last updated |
| last_updated_by | String | user to last update this instance |
Example Rules
Records Mgmt
Initialisation
switch
case:
- Pred: App in development && has no survey issued && is not euda
- State: INITIAL_SURVEY_ISSUED/IN_PROGRESS
- Side-effect: issueSurvey("INIT")
default:
- State: NOT_REQUIRED/PARKED
Transitions
HasRecords
- InitialState: INITIAL_SURVEY_ISSUED
- Pred: hasApprovedSurvey("INIT") && surveyAnswer("INIT", "Q1") == "HAS_RECORDS"
- State: DETAIL_SUREVY_ISSUED/IN_PROGRESS
- Side-effect: issueSurveyToAllEntitiesListedInSurvey("DETAIL", "INITIAL", "Q3") and updateAssessment("HAS_RECORDS", "YES")
DoesNotHaveRecords
- InitialState: INITIAL_SURVEY_ISSUED
- Pred: hasApprovedSurvey("INIT") && surveyAnswer("INIT", "Q1") == "DOES_NOT_HAVE_RECORDS"
- State: NO_RECORDS/COMPLETED
- Side-effect: updateAssessment("HAS_RECORDS", "NO")
DetailSurveyComplete
- InitialState: DETAIL_SUREVY_ISSUED
- Pred: hasApprovedSurvey("DETAIL")
- State: DETAIL_SURVEY_COMPLETED/COMPLETED
- Side-effect: updateRatingsFromSurveyQuestion("RECORDS", "12", "YES")
```
## Arch Gov
**Initialisation**
switch
case:
- Pred: CI is not retired and hasAssessment("ARCH_SCOPE") == 'IN_SCOPE' (from clarity checks)
- State: DISCOVERY_SURVEY_ISSUED/IN_PROGRESS
- Side-effect: issueSurvey("DISCOVERY")
default:
- State: NOT_REQUIRED/PARKED
**Transitions**
NoArchChange
- InitialState: DISCOVERY_SURVEY_ISSUED
- Pred: hasApprovedSurvey("DISCOVERY") && surveyAnswer("DISCOVERY", "Q1") == "No - No Architectural Change"
- State: DISCOVER_SURVEY_NO_ARCH_CHANGE/COMPLETED
- Side-effect: updateAssessment("ARCH_SCOPE", "DESCOPED")
ArchChange
- InitialState: DISCOVERY_SURVEY_ISSUED
- Pred: hasApprovedSurvey("DISCOVERY") && surveyAnswer("DISCOVERY", "Q1") == "Yes"
- State: ARCH_REVIEW_SURVEY_ISSUED/IN_PROGRESS
- Side-effect: issueSurvey("ARCH_REVIEW")
DiscoveryOverdue
- InitialState: DISCOVERY_SURVEY_ISSUED
- Pred: !hasApprovedSurvey("DISCOVERY") && monthsSinceLastUpdate > 6
- State: DISCOVERY_SURVEY_OVERDUE/IN_PROGRESS
- Side-effect: issueReminder()
ArchChangeFromOverdue
- InitialState: DISCOVERY_SURVEY_OVERDUE
- Pred: hasApprovedSurvey("DISCOVERY")
- State: DISCOVERY_SURVEY_ISSUED/IN_PROGRESS
ArchReviewCompleted
- InitialState: ARCH_REVIEW_SURVEY_ISSUED
- Pred: hasApprovedSurvey("ARCH_REVIEW")
- State: ARCH_REVIEW_COMPLETED/COMPLETED
- Side-effect: updateAssessment("ARCH_SCOPE", getValueFromSurveyQuestion("ARCH_REVIEW", "Q4"))
Arch Gov (using context variables)
Context
| name | lookupFn | example |
|---|---|---|
subject |
this is implicitly available | {extId: 'INV123', name: "Move to cloud", status: "ACTIVE" } |
scoping_assessment |
assessment("ARCH_IN_SCOPE") |
IN_SCOPE |
approval_survey |
survey_info("DISCOVERY") |
{ status: 'APPROVED', lastUpdate: '2021-09-15', ... } |
review_survey |
survey_info("ARCH_REVIEW") |
{ status: 'APPROVED', lastUpdate: '2021-12-25', ... } |
scope_answer |
survey_answer("DISCOVERY", "Q1") |
'Yes' |
survey_answers |
survey_answers("DISCOVERY") |
{Q1: 'Yes'', Q2: ['foo', 'baa'], ,,, } |
| name | lookupRef | example |
|---|---|---|
subject |
this is implicitly available | {extId: 'INV123', name: "Move to cloud", status: "ACTIVE" } |
scoping_assessment |
{kind: "assessment", ext: "ARCH_IN_SCOPE"} |
IN_SCOPE |
survey_answers.Q1.value === 'Yes' or scope_answer.value === 'Yes'
The above shows examples of direct vars (i.e. scope_answer) and aggregate vars (survey_answers).
Not sure which will be better yet, suspect direct vars will be clearer but the extra setup may offest this advantage.
There is also a tension around performance, aggregate vars are great if lots of parts are used, but are inefficient if only one part is needed.
In opposition direct vars are efficient if only one or two are needed, if lots are needed from the same parent entity then aggregates would probably be better. This may be mitigated by grouping similar lookupFns (like the approach taken in Report Grids)
Using the direct approach, the arch gov definition would look like:
Initialisation
switch
case:
- Pred: subject.status == 'ACTIVE' && scoping_assessment == 'IN_SCOPE' (from clarity checks)
- State: DISCOVERY_SURVEY_ISSUED/IN_PROGRESS
- Side-effect: issueSurvey("DISCOVERY")
default:
- State: NOT_REQUIRED/PARKED
Transitions
NoArchChange
- InitialState: DISCOVERY_SURVEY_ISSUED
- Pred: approval_survey.status == 'APPROVED' && scope_answer == "No - No Architectural Change"
- State: DISCOVER_SURVEY_NO_ARCH_CHANGE/COMPLETED
- Side-effect: updateAssessment("ARCH_SCOPE", "DESCOPED")
ArchChange
- InitialState: DISCOVERY_SURVEY_ISSUED
- Pred: approval_survey.status == 'APPROVED' && scope_answer == "Yes"
- State: ARCH_REVIEW_SURVEY_ISSUED/IN_PROGRESS
- Side-effect: issueSurvey("ARCH_REVIEW")
DiscoveryOverdue
- InitialState: DISCOVERY_SURVEY_ISSUED
- Pred: !approval_survey.status == 'APPROVED' && approval_survey.issuedOn > fromNow(6, "months")
- State: DISCOVERY_SURVEY_OVERDUE/IN_PROGRESS
- Side-effect: issueReminder()
ArchChangeFromOverdue
- InitialState: DISCOVERY_SURVEY_OVERDUE
- Pred: approval_survey.status == 'APPROVED'
- State: DISCOVERY_SURVEY_ISSUED/IN_PROGRESS
ArchReviewCompleted
- InitialState: ARCH_REVIEW_SURVEY_ISSUED
- Pred: review_survey.status == 'APPROVED'
- State: ARCH_REVIEW_COMPLETED/COMPLETED
- Side-effect: updateAssessment("ARCH_SCOPE", getValueFromSurveyQuestion("ARCH_REVIEW", "Q4"))
