Akka and Spark

[pisees]

Hi Richard,

Read your article with great interest and reaching hoping you can clarify how Akka/Actors can be used with Spark.

I am confused by articles about deprecation and removal of Akka in Spark documentation. Can Akka/Actors be used in a Spark solution still, or is it not recommended/deprecated.

• http://stackoverflow.com/questions/32045653/why-actorsystem-will-been-deprecated-in-spark
• Actor samples no longer in docs ** (not here) https://github.com/apache/spark/tree/v2.1.0/examples/src/main/scala/org/apache/spark/examples/streaming ** (but here) https://github.com/apache/spark/tree/branch-1.6/examples/src/main/scala/org/apache/spark/examples/streaming

Or is the issue how you create Akka solution? Don't integrate with Spark's Akka implementation ** class FeederActor extends Actor But create your own, as you have done in your article (https://www.spantree.net/blog/2016/10/18/spark-akka-sangria.html)

Reason asking, is investigating how to solve more complex workflows/dataflows that may lend themselves to be event driven. When a step finished, fires event (sends a message), and Actor could be listening for event, and execute the next step. Current simpler implementation in PoC is using Oozie.

And investigating whether this workflow/dataflow engine could be within Spark itself, for example with Akka/Actors, or perhaps external (Askaban, Airflow, other).

Thanks, Kurt