spark-csv icon indicating copy to clipboard operation
spark-csv copied to clipboard

Published 20 hours ago • verified publisher icon databricks

#60 provide type for custom columns

Open kostjas opened this issue 9 years ago • 6 comments

If you are not using userSchema by default all fields in csv file are assumed to be StringType. This commit adds possibility to setup types for fields which are not supposed to be as StringType.

Usage:

TestSQLContext.csvFile(carsTypedColumnsFile, fieldsTypeMap = Map("price" -> DoubleType, "new" -> BooleanType))

kostjas avatar Jun 30 '15 15:06 kostjas

+1

andrewrothstein avatar Jul 22 '15 14:07 andrewrothstein

@kostjas thanks for submitting this. Would you please rebase this? Also please add unit tests for CsvFastSuite as well.

falaki avatar Jul 22 '15 21:07 falaki

@falaki I rebased @kostjas branch and added a unit test to CsvFastSuite at andrewrothstein/spark-csv:master. Should I raise a separate pull request?

andrewrothstein avatar Jul 22 '15 22:07 andrewrothstein

If @kostjas is not interested in following up or does not have time for it, by all means. @kspitsyn What is your input?

falaki avatar Jul 22 '15 23:07 falaki

I rebased and squashed my commits into one. However if @andrewrothstein would like to accumulate fixes of several issues and push them simultaneously into master, I do not have objections.

kostjas avatar Jul 23 '15 16:07 kostjas

As I see pull request for #107 is ready :+1:

kostjas avatar Jul 23 '15 16:07 kostjas