csv-to-sql-insert icon indicating copy to clipboard operation
csv-to-sql-insert copied to clipboard

Published 20 hours ago • verified publisher icon gitdagray

Stream CSV file reading & SQL file writing

Open Vikram0118 opened this issue 1 year ago • 2 comments

The CSV file can get very huge. So, depending on the size of your CSV file, we might want to consider streaming the file instead of reading the entire content into memory.

Vikram0118 avatar Dec 22 '23 19:12 Vikram0118

This is a good idea!! 💯

gitdagray avatar Dec 22 '23 20:12 gitdagray

Processing them sequentially might lead to performance bottlenecks. Parallelizing the processing of multiple rows simultaneously can significantly improve overall performance. I once came across promise.all() for a project doing parallel reading. we can add that.

Vikram0118 avatar Dec 22 '23 20:12 Vikram0118