datawave icon indicating copy to clipboard operation
datawave copied to clipboard

Fix #1438 - optionally gather fields after ingest exception

Open matthpeterson opened this issue 2 years ago • 1 comments

matthpeterson avatar Feb 24 '22 21:02 matthpeterson

Multiple test cases were conducted using the quickstart json data:

  1. EVENT_DATE_MISSING added in JsonReader to every other record.
  2. getFields in JsonIngestHelper was made to throw an exception. JsonIngestHelper was also modified to implement SalvagedFields with a hardcoded multimap of mocked salvaged fields.
    Note both of the two test cases were done with throw-away code changes, made only for testing this PR.
  3. Regression case (no modifications, no errors)

Case 1 operated as it does today, including all the normally-ingested fields and the supplemental fields in the error table writes. The other half of the data was written to the shard table, as expected.

Case 2 operated with the new behavior: adding LOAD_DATE, ORIG_FILE, and salvaged fields to the error tables along with what it previously written.

Case 3 operated as expected, ingesting the data into the shard table.

matthpeterson avatar Mar 04 '22 21:03 matthpeterson