spark-structured-streaming-examples icon indicating copy to clipboard operation
spark-structured-streaming-examples copied to clipboard

Published 20 hours ago • verified publisher icon AndrewKuzmin
→

Metadata

Spark structured streaming examples with using of version 3.5.1

Spark Structured Streaming Examples

Spark structured streaming examples with using of version 3.4.0

Support matrix for joins in streaming queries

Left Input Right Input Join Type Example
Static Static All types TBD
Stream Static Inner TBD
Left Outer TBD
Right Outer Not supported
Full Outer Not supported
Left Semi TBD
Static Stream Inner TBD
Left Outer Not supported
Right Outer TBD
Full Outer Not supported
Left Semi Not supported
Stream Stream Inner ..streamstream.InnerJoinApp*, ..streamstream.InnerJoinWithWatermarkingApp*
Left Outer ..streamstream.LeftOuterJoinWithWatermarkingApp*
Right Outer TBD
Full Outer TBD
Left Semi TBD
*Base package: com.phylosoft.spark.learning.sql.streaming.operations.join

Use cases of processing modes (Triggers modes)

  1. Unspecified (default);
  2. Fixed interval micro-batches;
  3. One-time micro-batch (deprecated);
  4. Available-now micro-batch;
  5. Continuous with fixed checkpoint interval (experimental);

Optimizations

  1. Tungsten execution engine;
  2. Catalyst query optimizer;
  3. Cost-based optimizer;

Structured Sessionization

  1. KeyValueGroupedDataset.mapGroupsWithState;
  2. KeyValueGroupedDataset.flatMapGroupsWithState;

Links

  1. Structured Streaming Programming Guide;
  2. Stream-Stream Joins using Structured Streaming (Scala);
  3. Easy, Scalable, Fault-Tolerant Stream Processing with Structured Streaming in Apache Spark;
  4. Easy, Scalable, Fault-Tolerant Stream Processing with Structured Streaming in Apache Spark - continues;
  5. Deep Dive into Stateful Stream Processing in Structured Streaming;
  6. Monitoring Structured Streaming Applications Using Web UI;
  7. The Internals of Spark Structured Streaming;

← Metadata

25
Stars
14
Forks
Watchers

Owner

verified publisher icon AndrewKuzmin

Metadata

Spark structured streaming examples with using of version 3.5.1

Back