open-data-cnpj-import icon indicating copy to clipboard operation
open-data-cnpj-import copied to clipboard

Improvements to speed up download and insert routines

Open danielbohnrs opened this issue 2 years ago • 0 comments

  • bulk insert to mysql using load data from csv file, speeding up the entire process
  • added script to speed up download process using multiple connections (up to 1.2MB/s in my tests)

Obs: my current approach is: insert all data in temporary table and after it completes, rename currently table to correct one (this takes about 1 second ). This way we can always use the same database and tables to query the most recent data (no need to change the consumer application). In my tests, the load data process took about 1h.

danielbohnrs avatar Jun 01 '22 17:06 danielbohnrs