postgres_to_redshift
postgres_to_redshift copied to clipboard
print Redshift loading error
#3
- [x] print Redshift loading error as applicable.
- [x] fix a few lints (single quote etc) according to rubocop.
- [x] expose an optional flag to ignore error and keep loading other tables into Redshift. useful for debug.
Thanks! I left a few comments. Let me know what you think.
Also, what's an example of the error message being caught?
@toothrot thanks for reviewing. I've updated the pull request to improve the messaging. an example redshift loading error output:
INFO: Relation "some_table" already exists and will be skipped
Downloading some_table
Uploading some_table.1
Importing some_table
INFO: Table "some_table_updating" does not exist and will be skipped
ERROR: Load into table 'some_table' failed. Check 'stl_load_errors' system table for details.
ERROR: Last entry in Redshift's 'stl_load_errors' table:
userid: 100
slice: 0
tbl: 112357
starttime: 2016-07-19 20:07:36.182859
session: 4595
query: _n_
filename: s3://_some_file_.psv.gz.1
line_number: 11302
colname: body
type: varchar
col_length: 65535
position: 119
raw_line: a|b|c|...
raw_field_value: _some_raw_value_
err_code: 1220
err_reason: String contains invalid or unsupported UTF8 codepoints. Bad UTF8 hex sequence: ef bf bf (error 7)
INFO: Skipping 'some_table' and continuing on.
the last INFO line only prints if WARN_ON_LOADING_ERROR
is specified. for other exceptions, error and stacktrace will be printed.
@toothrot how's this one looking?
@toothrot ping?
I think I want to take another pass at this before merging it. I'll do that this week. Sorry for the delay.
@toothrot ping :)