postgres_to_redshift icon indicating copy to clipboard operation
postgres_to_redshift copied to clipboard

print Redshift loading error

Open jhuangtw opened this issue 8 years ago • 7 comments

#3

  • [x] print Redshift loading error as applicable.
  • [x] fix a few lints (single quote etc) according to rubocop.
  • [x] expose an optional flag to ignore error and keep loading other tables into Redshift. useful for debug.

jhuangtw avatar Jul 19 '16 17:07 jhuangtw

Thanks! I left a few comments. Let me know what you think.

toothrot avatar Jul 19 '16 18:07 toothrot

Also, what's an example of the error message being caught?

toothrot avatar Jul 19 '16 18:07 toothrot

@toothrot thanks for reviewing. I've updated the pull request to improve the messaging. an example redshift loading error output:

INFO:  Relation "some_table" already exists and will be skipped
Downloading some_table
Uploading some_table.1
Importing some_table
INFO:  Table "some_table_updating" does not exist and will be skipped
ERROR:  Load into table 'some_table' failed.  Check 'stl_load_errors' system table for details.
ERROR:  Last entry in Redshift's 'stl_load_errors' table:
    userid: 100
    slice: 0
    tbl: 112357
    starttime: 2016-07-19 20:07:36.182859
    session: 4595
    query: _n_
    filename: s3://_some_file_.psv.gz.1                                                                                                                                                                                                        
    line_number: 11302
    colname: body                                                                                                                           
    type: varchar   
    col_length: 65535     
    position: 119
    raw_line: a|b|c|...
    raw_field_value: _some_raw_value_
    err_code: 1220
    err_reason: String contains invalid or unsupported UTF8 codepoints. Bad UTF8 hex sequence: ef bf bf (error 7)   

INFO:  Skipping 'some_table' and continuing on.

the last INFO line only prints if WARN_ON_LOADING_ERROR is specified. for other exceptions, error and stacktrace will be printed.

jhuangtw avatar Jul 19 '16 20:07 jhuangtw

@toothrot how's this one looking?

jhuangtw avatar Jul 24 '16 19:07 jhuangtw

@toothrot ping?

jhuangtw avatar Aug 09 '16 06:08 jhuangtw

I think I want to take another pass at this before merging it. I'll do that this week. Sorry for the delay.

toothrot avatar Aug 09 '16 15:08 toothrot

@toothrot ping :)

jhuangtw avatar Aug 29 '16 21:08 jhuangtw