incubator-livy icon indicating copy to clipboard operation
incubator-livy copied to clipboard

The SparkStreaming operator fails to execute

Open zenvzenv opened this issue 2 years ago • 0 comments

kafka as SparkStreaming input and output

  1. Use spark.readStream.format("kafka") read kafka data and decode binary data to string
  2. Use df.map(_.Seq.foldLeft(""))(_ + separtor + _).writeStream("kafka") output data to kafka
  3. If I fail to output to kafka, then no matter how I change the kafka topic later, the stream computing will fail,ArrayIndexOutOfBoundsException: 1 exception will report.If I only output to the console there will be no error
  4. If I run the same code snippet directly in spark-shell without using livy, the effect is the same as 3

zenvzenv avatar Aug 17 '23 08:08 zenvzenv