spring-cloud-dataflow icon indicating copy to clipboard operation
spring-cloud-dataflow copied to clipboard

Published 20 hours ago • verified publisher icon spring-cloud

Can scdf work with spark?

Open xgxizz opened this issue 3 years ago • 1 comments

Can this tool work with spark? If so, can you provide a simple example as a reference.

xgxizz avatar Aug 25 '22 06:08 xgxizz

Hi. That is a pretty general question, what use-case did you have in mind? The precursor to Spring Cloud Data Flow, Spring XD, has some integration but it was not used much so we did not continue to maintain that functionality. See https://docs.spring.io/spring-xd/docs/current/reference/html/#spark-app Other common use-cases that I can think of off the top of my head would be to use SCDF to store data in some format/database/system from which a Spark Job can load it up into a dataframe and do the processing in Spark.

markpollack avatar Aug 30 '22 14:08 markpollack

Closing the issue.

markpollack avatar Sep 22 '22 14:09 markpollack