CsvReader icon indicating copy to clipboard operation
CsvReader copied to clipboard

[Question] Faster way to read the csv

Open JHuenerberg opened this issue 8 years ago • 2 comments

Hey guys and thanks for the awesome work!

Could be a ReadAllLines() method, then run a Parallel.For loop to process all the lines in parallel way faster than the used single StreamReader ?

JHuenerberg avatar Jan 30 '17 22:01 JHuenerberg

Why don't you try - there's benchmarks for the existing code so you can compare performance on your machine,.

phatcher avatar Feb 03 '17 11:02 phatcher

  1. This doesn't work for embedded line-breaks
  2. This requires buffering the entire file in memory before processing it

Even for the cases where this would work, it would cause high GC pressure and promote a lot of Gen2 objects for all but the smallest of files that would not benefit much or at all of concurrency.

bcronce avatar Jun 15 '18 17:06 bcronce