sbarnoud

Hi Sorry but metrics reporting doesn't work for me. I already configure my job to have them, and if i change my sink to parquet on the same stream, i...

https://stackoverflow.com/questions/48466019/number-of-input-rows-in-spark-structured-streaming-with-custom-sink That's the reason why metrics are not updated.

This code works, and avoids the current comment "allows us to do magic" ;-): ``` class HBaseStreamSink(sqlContext: SQLContext,options: Map[String, String]) extends Sink with Logging { val defaultFormat = "org.apache.spark.sql.execution.datasources.hbase" val...

Hi, I have on my own version added the support of "short names" -) shc -) shcstream Could you validate those short names ? core/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister ``` org.apache.spark.sql.execution.datasources.hbase.DefaultSource org.apache.spark.sql.execution.streaming.HBaseStreamSinkProvider ``` org/apache/spark/sql/execution/datasources/hbase/HBaseRelation.scala...

No, i let you decide the names but i sent you the correct code. Both batch and streaming short names must be defined and different and both classes must be...

For the remaining problem with the order of the iterator, notice that I'm running Java8 and that your tests seem to be executed in Java7

Looks like you forget to connect to HBase Configuration hadoopConf = HBaseConfiguration.create(); Connection connection = ConnectionFactory.createConnection(hadoopConf); See testBulkPut in https://github.com/apache/hbase/blob/master/hbase-spark/src/test/java/org/apache/hadoop/hbase/spark/TestJavaHBaseContext.java

I have exactly the same issue using Kafka API of EventHub (but after some minutes to hours). Did you try to do the same with this connector (which uses EventHub...

So in our use case, Azure support identify that the root cause is not EventHub but our network setting. To summarize, we have a Hub&Spoke architecture where BGP announces of...

azure-messaging-eventhubs is for .Net. For Java (which is the case of this connector), it stills azure-eventhubs https://learn.microsoft.com/en-us/java/api/com.microsoft.azure.eventhubs?view=azure-java-stable But indeed, your right, It seems that documentation for Java is also not...