shc
shc copied to clipboard
hortonworks-spark
Unable to save data to Hbase
I am getting the Following error while updating a hbase table Exception in thread "main" org.apache.spark.sql.execution.datasources.hbase.InvalidRegionNumberException: Number of regions specified for new table must be greater than 3. Df.write.options(Map(HBaseTableCatalog.tableCatalog -> catalog)).format("org.apache.spark.sql.execution.datasources.hbase").save()
If i try modifying Df.write.options(Map(HBaseTableCatalog.tableCatalog -> catalog,HBaseTableCatalog.newTable -> "5")) .format("org.apache.spark.sql.execution.datasources.hbase").save() it works fine.
If it is already precreated table why i need to specify the new table option with regions specified.
Please help on this.
You may want to check this discussion thread: https://github.com/hortonworks-spark/shc/issues/76
Write to HBase table to populate data
sc.parallelize(data).toDF.write.options( Map(HBaseTableCatalog.tableCatalog -> catalog, HBaseTableCatalog.newTable -> "5")) .format("org.apache.spark.sql.execution.datasources.hbase") .save() Given a DataFrame with specified schema, above will create an HBase table with 5 regions and save the DataFrame inside. Note that if HBaseTableCatalog.newTable is not specified, the table has to be pre-created.
My question is as per the statement the hbase table is already precreated,why is it asking to specify the number of regions.
Hi, Is there an answer for above question. @ArunChandrasekar were you able to solve this ?
Hi, Is there an answer for above question. @ArunChandrasekar were you able to solve this ?
No I am not getting appropriate answer for this.I am trying newapihadooprdd which fits for my requirement.
Hi, there are some updates about this issue? We need to write to an existing HBase table, and there is no reason why we need to pass the number of regions... Thanks.
