open-data-cnpj-import
open-data-cnpj-import copied to clipboard
Improvements to speed up download and insert routines
- bulk insert to mysql using load data from csv file, speeding up the entire process
- added script to speed up download process using multiple connections (up to 1.2MB/s in my tests)
Obs: my current approach is: insert all data in temporary table and after it completes, rename currently table to correct one (this takes about 1 second ). This way we can always use the same database and tables to query the most recent data (no need to change the consumer application). In my tests, the load data process took about 1h.