datapusher-plus
datapusher-plus copied to clipboard
Use `psql /copy` instead of `psycopg2 copy_expert()`
psql /copy
is MUCH faster than psycopg2 copy_expert()
Inserting a 50mb CSV file took all of 2 seconds for psql /copy
and 215 seconds for psycopg2 copy_expert
- 100x faster, 2 orders of magnitude faster on the same machine!
We're calling several CLI commands already - qsv
, iconv
, file
& uchardet
- might as well add psql
to the mix.