Event driven spark application

[benczekristof]

Dear Maintainers,

I would like to ask a question a challenge regarding the usage of the spark-k8s operator: We would like to trigger the execution of a SparkApplication right after an event(possibly with a Rest call), but right now the only way i can think about is via creating a web service application with a service account that has the necessary permissions to call the K8S api to create SparkApplication objects.
Is there any other more convenient/secure way to trigger the execution of a spark application?

[jdonnelly-apixio]

We use airflow to do this. Airflow has an operator that works with the spark on k8s operator that can be used to define dags that kick off spark applications and airflow has a convenient api for triggering dags.