fastcsv icon indicating copy to clipboard operation
fastcsv copied to clipboard

fastcsv breaks with StringIO containing new lines near empty fields near BUFSIZE boundaries

Open thomasdziedzic-census opened this issue 2 years ago • 1 comments

Hi, it seems like when there is a new line after a blank field near the BUFSIZE, passing in a StringIO object is broken, but a String seems to work. I'm specifically referring to the BUFSIZE at https://github.com/jpmckinney/fastcsv/blob/master/ext/fastcsv/fastcsv.rl#L177

using fastcsv 0.0.6 and ruby 2.7.6

minimal reproducible test case:

require 'fastcsv'

csv = ''

bufsize = 2 ** 14
rowsize = 2 ** 3
row = (',' * (rowsize - 1)) + "\n"
num_rows = bufsize / rowsize
num_rows.times do
  csv << row
end

FastCSV.raw_parse(StringIO.new(csv)) do |row|
end

output:

% bundle exec ruby fastcsv_bug.rb
Traceback (most recent call last):
        2: from fastcsv_bug.rb:13:in `<main>'
        1: from /Users/thomas/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/fastcsv-0.0.6/lib/fastcsv.rb:8:in `raw_parse'
/Users/thomas/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/fastcsv-0.0.6/lib/fastcsv.rb:8:in `raw_parse': Unquoted fields do not allow \\r or \\n (line 2048). (FastCSV::MalformedCSVError)

thomasdziedzic-census avatar Jun 17 '22 12:06 thomasdziedzic-census

I can reproduce the error. I haven't been able to determine why the error occurs (it's been a while since I wrote Ragel.)

jpmckinney avatar Jun 17 '22 17:06 jpmckinney