spanner-migration-tool
spanner-migration-tool copied to clipboard
Improve error messages during data conversion.
In general, HarbourBridge provides a summary of errors and unusual conditions as part of the 'Unexpected Conditions" section of report.txt. However, this section does not including errors encountered while writing rows to Spanner.
Note that such errors are accumulated in BatchWriter and are available via the Errors() method, but HarbourBridge does not call Error and so these errors are essentially dropped.
There are several options to improve this:
a) Improve error handling in data.go so that more errors are caught before we call Spanner (e.g. we could catch common errors like non-matching column/value counts, and field size mismatches). This has other benefits: we can provide more succinct error messages and avoid batch retries/splits.
b) Change report.go to include errors from BatchWriter (requires some minor refactoring of main.go).