datawave
datawave copied to clipboard
Fix #1438 - optionally gather fields after ingest exception
Multiple test cases were conducted using the quickstart json data:
- EVENT_DATE_MISSING added in JsonReader to every other record.
- getFields in JsonIngestHelper was made to throw an exception. JsonIngestHelper was also modified to implement SalvagedFields with a hardcoded multimap of mocked salvaged fields.
Note both of the two test cases were done with throw-away code changes, made only for testing this PR. - Regression case (no modifications, no errors)
Case 1 operated as it does today, including all the normally-ingested fields and the supplemental fields in the error table writes. The other half of the data was written to the shard table, as expected.
Case 2 operated with the new behavior: adding LOAD_DATE, ORIG_FILE, and salvaged fields to the error tables along with what it previously written.
Case 3 operated as expected, ingesting the data into the shard table.