data.gov icon indicating copy to clipboard operation
data.gov copied to clipboard

Update DB after MDTranslator Transform

Open btylerburton opened this issue 1 year ago • 1 comments

User Story

In order to keep an accurate count of the state of our datasets, datagovteam wants to update the Harvesting database with the results of the transformation step by MDTranslator.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

  • [ ] GIVEN a record is marked to be transformed by MDTranslator AND that operation is successful THEN we should log that success AND that dataset should be passed along to the validate operation

  • [ ] GIVEN a record is marked to be transformed by MDTranslator AND that operation is unsuccessful THEN we should log that failure AND we should create a new error in the harvest_error table AND we should mark that dataset "state: error" in the harvest_records db

Background

[Any helpful contextual notes or links to artifacts/evidence, if needed]

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

Sketch

  • [ ] Loop over records to be transformed
  • [ ] Log the result of the transform operation
  • [ ] Create an error if necessary at runtime
  • [ ] Push the dataset into a failed_transform list if it fails to transform

btylerburton avatar May 20 '24 18:05 btylerburton

Tyler will update AC.

gujral-rei avatar May 23 '24 20:05 gujral-rei