datapusher-plus icon indicating copy to clipboard operation
datapusher-plus copied to clipboard

Use `psql /copy` instead of `psycopg2 copy_expert()`

Open jqnatividad opened this issue 1 year ago • 1 comments

psql /copy is MUCH faster than psycopg2 copy_expert()

Inserting a 50mb CSV file took all of 2 seconds for psql /copy and 215 seconds for psycopg2 copy_expert - 100x faster, 2 orders of magnitude faster on the same machine!

We're calling several CLI commands already - qsv, iconv, file & uchardet - might as well add psql to the mix.

jqnatividad avatar Jan 24 '24 15:01 jqnatividad