Make CsvFileReader.readNext() private

[doyaaaaaken]

Problem

If we use readNext method, some options (e.g. excessFieldsRowBehaviour option) are bypassed and invalid.

There does not appear to be a use case for this method, so we are considering making it a private method. If you have feedback, please comment.

[bobbyphilip]

Would be nice to have this (or an equivalent) to be able to just read the header row of a CSV file ( Maybe to just get the names of the columns)

[ipat500]

I think it would be good to have access to basic CSV parser functionality, without the line length checks or header parsing logic. As a fallback in case you need to parse very unusual CSVs. Or to parse malformed CSVs for error reporting or other purposes.

If a file contains blank rows, or rows with differing numbers of fields, then it is still possible to load these files into Excel. Spreadsheet or generic CSV parsing software does not care about the number of fields, or headers, it only cares about the most basic structure of a CSV. Even a malformed CSV with blank rows or rows with a different number of fields will be laid out in table form with row and column numbers. This type of software consistently puts cells in the same row and column numbers, regardless of the vendor. I have tested this with Excel, Libre Office and Modern CSV and they all give consistent row numbers for files with blank rows, differing numbers of fields and fields containing newlines.

Using CSVFileReader.readAllAsSequence() it is not possible to replicate the lenient parsing behavior of those types of tools. It is not even possible to consistently determine which row numbers they assign to specific rows. However using CSVFileReader.readNext() these things are possible for any file.

insufficientFieldsRowBehaviour, excessFieldsRowBehaviour and skipEmptyLine allow for a bit of wiggle room in some use-cases, but do not allow the checks to be bypassed entirely. For example it is not possible to parse a file with blank rows unless skipEmptyLine = true. However if this is enabled then readAllAsSequence() will not return those rows at all. This is an issue if you are trying to report the row number in the file correctly since you can not know if the line exists or not. Rows with differing numbers of fields have other issues. With readNext() you can always get the correct row number in the file, and the content, regardless of how many fields it has or if it is blank.

With CSVFileReader.readNext() it is possible to replicate the lenient parsing behavior of spreadsheet software, or to report errors with row numbers that are consistent with spreadsheet software. Without readNext() these things are no longer possible.

I do agree that the behavior of readNext() may be confusing. A user might assume that readNext() behaves the same as readAllAsSequence().iterator().next() and thus respects insufficientFieldsRowBehaviour, excessFieldsRowBehaviour and skipEmptyLine. It may be confusing and unexpected that it does not. As an alternative to readNext() it might be better to add a function readAllAsSequenceRaw() or a separate class which provides this raw row read functionality. Or adding a config option which bypasses the field number checks entirely.

[doyaaaaaken]

Thank you for your very useful input. I completely agree with your opinion. I'll consider the future direction of this feature.

[Thijsiez]

I'm currently using readNext() in 2 ways;

• to read the headers only
• to read the first X rows to grab a preview of the CSV file, without needing any of the checks that happen when reading the whole file

If readNext() were to be removed, I'd love to have the option to call something like readColumnNames() or similar, which ideally also checks for duplicates. Even better, being able to call something like CsvReader.readColumnNames(File) would likely limit misuse and be prettier than having to open(File) first.

For my second use case, there currently are only utility methods to readAll in different ways, but being able to read the first X rows including headers and the full set of checks used would be amazing to provide previews!! Maybe something like CsvReader.readWithHeader(File, Int? = null) where, if passed, the integer value would specify how many rows to read before returning :)

[rverma-dev]

It seems readNext() is line by line read and can work in very less memory which readAllSequence() will force loading all the csv in memory first. What is the alternate if memory is constraint or we are suppose to process very large csvs

[abdallahaymaan]

I am using readNext() because I want to skip rows that fail (due to escapeChar or delimiter issues or anything else) and continue reading the rest of the lines. I do so be try and catch when reading each line of the csv. With readAllSequence() I get an error for the whole file and can't just neglect failed rows and read others. If it's possible with some other way than readNext() please let me know.