CDCgov
Wisconsin - UP (Shadow) Migration - Validate Test Data
[STLT] = name of STLT
User Story:
As a developer, I want [STLT] to be able to receive COVID data in the exact same format and contents from the Universal Pipeline, as it was from the COVID Pipeline. This will allow us to migrate the receiver without sending any test messages, enabling our decommission of the COVID Pipeline sooner.
Description/Use Case
[STLT] is already receiving data from the covid pipeline. They will be onboarded to the UP where they will be receiving COVID data from SimpleReport. We need to make sure the message we generate from the UP passes the STLTs validation and matches the output from the COVID pipeline exactly. We can use SimpleReport to submit test messages through the UP for internal checking, but we will not be sending test messages to the [STLT] testing environment.
Risks/Impacts/Considerations
- Since we will not be confirming test message contents with the STLT, we need to ensure internally that the messages meet STLT requirements before moving to production.
Dev Notes:
Github Documentation: LINK
-
CSV Upload messages are found here: https://docs.google.com/spreadsheets/d/1DtmP7llf9KFDBKsu89I77eSyiAVykk8TTJ2G6gJy0hw/edit?gid=1333277098#gid=1333277098
- They may need to be recreated by using this google sheet and converting to CSV: https://docs.google.com/spreadsheets/d/1DtmP7llf9KFDBKsu89I77eSyiAVykk8TTJ2G6gJy0hw/edit?gid=551002913#gid=551002913
-
[ ] Ensure that Steps 1-4 have already been completed from github documentation
-
[ ] Grab CSV files from message test bank See: https://app.zenhub.com/workspaces/engagement-6166edbd409257001e09f1b2/issues/gh/cdcgov/prime-reportstream/16831
- [ ] Messages may need to be re-created if SimpleReport updated their messages
-
[ ] Fetch covid pipeline settings for [STLT] from production using
./prime multiple-settingsand load the settings locally -
[ ] Run
./prime data --input {input file} --input-schema {primedatainput/pdi-covid-19 | upload-covid-19} --output {output file} --route-to {orgName.receiverName}against all CSV files using receiver schema and generate HL7 files- [ ] Input schema: use primedatainput/pdi-covid-19 for CSV upload messages and upload-covid-19 for single entry messages
- [ ] Route To: use covid pipeline receiver name and org name retrieved from previous step.
-
[ ] Run
./prime fhirdata --input-file {input file} -s classpath:/metadata/fhir_transforms/senders/SimpleReport/simple-report-sender-transform.yml --output-format HL7 --output-file {output file} --receiver-name={receiver name}against all fhir bundles using receiver UP receiver settings- [ ] Input file: UP message matching CSV from covid pipeline
- [ ] Receiver Name: UP receiver with receiver schema transforms and any necessary enrichments
-
[ ] Run comparison between the CP and UP HL7 messages
- [ ] Review output
-
[ ] Verify there are no new differences and that SimpleReport has addressed these issues:
- [ ] Single Entry
- [ ] Code and description for symptom onset date AOE match
- [ ] Bulk Upload
- [ ] Positive/Negative results codes and descriptions match
- [ ] Corrected status shows up on all AOE entries
- [ ] Verify no symptomatic_for_disease AOE is generated in the UP when this value is empty in the CSV
- [ ] Single Entry
-
[ ] Document differences and review with team
Acceptance Criteria
- [ ] Provide feedback for any process improvements for follow up shadow migrations( Review during Eng Sync)
- [ ] Document differences between CP and UP COVID messages
- [ ] Review documented differences with the team
- [ ] Get approval from Tech lead and Product to migrate receiver to production
- [ ] When finished, tag DAvid, Chris, Victor, and Glenn in a slack post stating:
[STLT] "Shadow Migration" is complete and is ready to be turned on in production
Connection Type in Prod: [REST, SFTP, SOAP, etc]
Positives set up to receive: COVID
Negatives set up to receive: [List of Conditions]
Format receiving: [HL7 or FHIR]
Script Attached. It will creat an Hl7 file from the Covid pipeline and an hl7 from the UP pipeline, from the csvs files created fron the pipelines. It will then compare the two hl7 files and see if there are any differences . It will a create and output log files with the differences. validate_test_data.txt
Just rename from txt to sh
