Lakshmanan Ramakrishnan comments

Repositories
Issues
Comments

Results 1 comments of


                                            Lakshmanan Ramakrishnan

spark-avro 2.0.1 generates strange schema (spark-avro 1.0.0 is fine)

I believe the issue is DefaultSource extends HadoopFsRelationProvider and in spark https://github.com/apache/spark/blob/branch-1.6/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ResolvedDataSource.scala#L112 For all datasources extending from HadoopFsRelationProvider, all columns are nullable. This looks to be a very bad issue...