Rohit Karlupia

@prateekgupta I suspect that this is something to do with how the filesystem is being initialised, very likely missing some mandatory configuration. We are using `new Configuration()` which should pick...

@gyjwork what is the spark version you are using? And what specific distribution, if any?

@normalscene The negative numbers can show up for various reasons. One of the things we try to do is to find out how much time executors were busy vs when...

@normalscene Your will need to add the following config options to your spark-submit. spark.eventLog.enabled (true) spark.eventLog.dir (s3:///some/where/) After the application is complete, event log file will be available in the...

@normalscene Not really much aware of the internal workings of dataproc. If you have access to history server, newer versions allow downloading this file directly from the history server UI.

Thanks for raising this issue. I will check and revert back shortly. Are you using dynamic allocation / autoscaling of executors?

Thanks for bringing this up @dominikabasaj. This is definitely on the radar and we will be adding support for Streaming. I will encourage you to wear a PM hat and...

@dominikabasaj Here is one way to get it working with streaming job. I haven't tried it with streaming yet. Let me know if this serves your purpose. 1.Start your application...

Here is one way to get it working with streaming job. I haven't tried it with streaming yet. Let me know if this serves your purpose. 1. Start your application...

@akumarb2010 This is a good start! I believe one of the key questions for streaming jobs would be: if my input data rate increases, can I still meet the SLA...