Spark-MongoDB
Spark-MongoDB copied to clipboard
Cannot parse splitSize properly
If I use the syntax provided by Spark-SQL as:
val parameters = Map(
"host" -> "example.com",
"database" -> "example",
"collection" -> "test",
"schema_samplingRatio" -> "0.0001",
"splitSize" -> "20"
)
then, the splitSize can not be parsed properly in the file com.qiyi.mongo.datasource.mongodb.partitioner.MongodbPartitioner.
It is caused by partially parsing parameter when initializing DataSouce.
I work out a patch for this issue, will pull it right now.