fastcsv

fastcsv copied to clipboard

fastcsv breaks with StringIO containing new lines near empty fields near BUFSIZE boundaries

1

Hi, it seems like when there is a new line after a blank field near the BUFSIZE, passing in a StringIO object is broken, but a String seems to work....

thomasdziedzic-census

Allow null / zero bytes in CSVs

2

When an input CSV contains the 0 / null byte, FastCSV emits two rows for that row, both "incorrect": ```ruby null_csv = "one,two\nthr\00ee,four" FastCSV.raw_parse(null_csv) { |row| p row } ```...

bradleybuda

Like in https://github.com/sj26/lenientcsv HippieCSV just replaces an escaped quote character with a doubled quote character, before parsing. https://github.com/intercom/hippie_csv/blob/91f247ffaa45ffb15798bfd3637fea73434762e1/lib/hippie_csv/support.rb#L52

jpmckinney

Ruby's CSV library has a single test for this: `test_leading_empty_fields_with_multibyte_col_sep_bug_fix` in `test_features.rb`. Ragel inlines the `when` block in an `if` statement; if the solution requires complex code, we may want...

jpmckinney

:row_sep support is more complicated than `:quote_char` and `:col_sep` support, because it is frequently multi-byte (e.g. `\r\n`) and needs to support the `:auto` option. May be easier to implement once...

jpmckinney

Options: - Use Ragel's error actions - Switch to row-by-row parser

jpmckinney

Replaces #16 - same basic idea. We've been running this in production at Census for about 4 weeks now (we parse tens of thousands of CSVs every day from a...

bradleybuda