visidata icon indicating copy to clipboard operation
visidata copied to clipboard

[csv-] use newline='' for opening files

Open midichef opened this issue 3 months ago • 1 comments

The csv loader should use newline=''. According to the csv library documentation:

"If newline='' is not specified, newlines embedded inside quoted fields will not be interpreted correctly [...] It should always be safe to specify newline='', since the csv module does its own (universal) newline handling.

newline='' is already used for saving CSV files (https://github.com/saulpw/visidata/issues/1362), added by 7a5e8341237c539ff9b8bdd51ff568d822e015b8.

An example file that demonstrates the difference: newlines.csv In vd newlines.csv, the third row, second column, loads as "\n" (on my Ubuntu system where newline is "\n". On Windows it would load as "\r\n", I think.). After the patch it loads as "\r\n".

I'm making this PR a draft, pending others' input, since I'm not well-versed in parsing CSV files.

Is there a canonical test suite of CSV files for parsing? I'd prefer to test this change before it's deployed.

midichef avatar Apr 07 '24 00:04 midichef