spark-csv

spark-csv copied to clipboard

Fixed width support.

6

Adds Relation, LineReader and BulkReader traits to avoid duplicated code. Largely derived from https://github.com/quartethealth/spark-csv and https://github.com/quartethealth/spark-fixedwidth. This is in response to the following PR (created by @blrnw3) being closed without...

etspaceman

added comments for csv file paths

1

Added the comments for csv file paths

msathiyarajan

Incorrectly checking ignoreLeadingWhiteSpace twice

1

omervk

[SPARK-16512] Adding a insertNullOnErrors option.

1

This is the change that allows an option to render errors when parsing such as number format exceptions as nulls. It was in this pull request, https://github.com/databricks/spark-csv/pull/298 but I thought...

rachelwarren

Upgraded to spark 2.0

3

davidcrossland

parsing options and serializing arrays

8

several parsing options are added. they are organized in classes because there are many of them. a "text" based API to configure options is provided. another feature is the ability...

mohitjaggi

report the error as well as the line that caused it

3

I don't know Scala (at all!) so there's almost certainly cleaner ways - my apologies. The logging at the moment is sometimes unhelpful as it's hard to see the real...

abridgett

Refactoring CsvParser.

1

For the context and discussion on this, please refer to https://github.com/databricks/spark-csv/pull/244.

tanwanirahul

Configurable null values

8

There's datasets where each column has it's own marker for missing values. spark-csv assumes only empty string for missing values. To avoid additional data transformation and saving on user's side...

petro-rudenko

#60 provide type for custom columns

6

If you are not using userSchema by default all fields in csv file are assumed to be StringType. This commit adds possibility to setup types for fields which are not...

kostjas