pgloader missing records in rejected dat file, seeing - BUG: failed to retry-batch: Invalid index_2

[gutpapr]

1. I am trying to upload a csv file to pgloader which has 1600 , 1200 got uploaded but out .dat file has only 300, where are other 100 records

example run -- ./pgloader -L my.log -S summary.csv -D ./ load_ALL.pgl

load_ALL.pgl is like --

LOAD CSV FROM '/home/legerity/mad_file_stage/inbound/my-data.csv'

INTO postgresql://user:password@dbhost:5432/db_name?tablename=my_tab

WITH truncate, fields terminated by ',', on error resume next, batch rows = 2 ;

• My postgresql destination table has some referential integrity constraint on one of the column, so as expected around 200+ records failed to load with the error in log, but in the output file I see only around100 records reported

• I was expecting that since it is retying in batches until it eliminates all rejected record I will get the .dat file with all rejected record but that is not happening.

• However number of records which are uploaded in database is correct and matches to the expectation.

• rejected .dat file has more record if i use the ' batch rows = 2' instead of 'batch rows = 200' but it is still not having all the rejected records.

FYI - version is //// I believe it is the latest

• → pgloader --version pgloader version "3.6.2" compiled with SBCL 2.0.1.debian

  2. But I see that some errors in log ---- ERROR BUG: failed to retry-batch: Invalid index 2 for (SIMPLE-VECTOR 2), should be a non-negative integer below 2. Not sure but looks lie retry is not working fine here ??

DETAIL: Key (account_class)=(CV) is not present in table "ref_account_class". 2021-10-24T00:08:16.793000Z ERROR PostgreSQL [""my_schema"."tablename""] Database error 23503: insert or update on table "tablename" violates foreign key constraint "fk_account_class" DETAIL: Key (account_class)=(CV) is not present in table "ref_account_class". 2021-10-24T00:08:16.812000Z ERROR BUG: failed to retry-batch: Invalid index 2 for (SIMPLE-VECTOR 2), should be a non-negative integer below 2. 2021-10-24T00:08:16.866000Z ERROR PostgreSQL [""my_schema"."tablename""] Database error 23503: insert or update on table "tablename" violates foreign key constraint "fk_account_class" DETAIL: Key (account_class)=(CV) is not present in table "ref_account_class". 2021-10-24T00:08:16.884000Z ERROR BUG: failed to retry-batch: Invalid index 2 for (SIMPLE-VECTOR 2), should be a non-negative integer below 2. 2021-10-24T00:08:16.956000Z ERROR PostgreSQL [""my_schema"."tablename""] Database error 23503: insert or update on table "tablename" violates foreign key constraint "fk_account_class" DETAIL: Key (account_class)=(CV) is not present in table "ref_account_class". 2021-10-24T00:08:17.050000Z ERROR PostgreSQL [""my_schema"."tablename""] Database error 23503: insert or update on table "tablename" violates foreign key constraint "fk_account_class" DETAIL: Key (account_class)=(CV) is not present in table "ref_account_class". 2021-10-24T00:08:17.134000Z ERROR PostgreSQL [""my_schema"."tablename""] Database error 23503: insert or update on table "tablename" violates foreign key constraint "fk_account_class" DETAIL: Key (account_class)=(CV) is not present in table "ref_account_class". 2021-10-24T00:08:17.154000Z ERROR BUG: failed to retry-batch: Invalid index 2 for (SIMPLE-VECTOR 2), should be a non-negative integer below 2. 2021-10-24T00:08:17.246000Z ERROR PostgreSQL [""my_schema"."tablename""] Database error 23503: insert or update on table "tablename" violates foreign key constraint "fk_account_class" DETAIL: Key (account_class)=(CV) is not present in table "ref_account_class". 2021-10-24T00:08:17.351000Z ERROR PostgreSQL [""my_schema"."tablename""] Database error 23503: insert or update on table "tablename" violates foreign key constraint "fk_account_class" DETAIL: Key (account_class)=(CV) is not present in table "ref_account_class". 2021-10-24T00:08:17.480000Z ERROR PostgreSQL [""my_schema"."tablename""] Database error 23503: insert or update on table "tablename" violates foreign key constraint "fk_account_class"

3. record in CSV file uploaded > record uploaded + records in rejected / out .dat file