Need a long running Spark job or a simulation involving Riosock to test memory leak

[hebinhuang]

Use a long running Spark job or a simulation involving Riosock to test memory leak

[qualiu]

@skaarthik @hebinhuang : Seems we can use the testMobius with Socket or Kafka streaming. I can provide some usage example and help if you like.

[qualiu]

you can use one command line to test, more tool and usage follow RIOSocket test and with memory test brief the command lines as following :

• Socket : scripts\memory\local-Riosocket-test-by-socket.bat 1
• Kafka : scripts\memory\local-Riosocket-kafka-test.bat 1

[hebinhuang]

There is an issue in testMobius with RIOSocket. the issue is still in debugging.

[hebinhuang]

There is a potential issue in Mobius that cannot keep testMobius with socket on long running.

[skaarthik]

@hebinhuang - can you elaborate on your last comment and also comment if this impacts M3?

[hebinhuang]

@skaarthik , The issue is happened on the testKeyValueStream with a socket data source. The testKeyValueStream starts a loop to test KeyValueStream, the StreamingContext will be started and stopped in each loop iteration. After about 7~10 iterations, the testKeyValueStream gets stuck. The scenario in the testKeyValueStream is not a typical scenario in Spark applications. A typical long running scenario is that a spark application starts only one StreamingContext and stop it when the application exists. So the issue does not impact M3.

Currently, @qualiu is working on the investigation.