[LIVY-870] Adding changes to provide Spark config using --properties-file

[akshatb1]

What changes were proposed in this pull request?

Currently for batch jobs, individual Spark configs are added as --conf parameter in Spark submit command line. If there are many Spark configurations added in Livy APIs, it might increase the length of command line and in some cases might cause issues due to limit on command line length (Windows has 8191 character length limit on the command line length.).

In this PR, modifying the behavior to merge all the configurations into a temporary properties file and adding that as --properties-file opt in spark-submit command

JIRA: https://issues.apache.org/jira/browse/LIVY-870

How was this patch tested?

Verified Spark job submission in a local Yarn cluster. Checked in the logs that all configurations are merged to a properties file which is added as an opt in spark-submit command. Please see the logs below.

21/10/12 12:01:15 INFO SparkProcessBuilder: Merged Spark config and generated file: D:\Data\Livy0.8\sparkConfig7641984478423208311.properties
21/10/12 12:01:15 INFO SparkProcessBuilder: Running 'D:/Spark/\bin\spark-submit.cmd' '--name' 'SparkBatchJobTest-14' '--class' 'WordCountJob' '--properties-file' 'D:\Data\Livy0.8\sparkConfig7641984478423208311.properties' 'file:///D:/Spark/examples/jars/default_artifact.jar' 'D:/Test.txt'

[leesf]

@akshatb1 would you please rebase to latest master to fix the CI failure?

[akshatb1]

@akshatb1 would you please rebase to latest master to fix the CI failure?

Thanks @leesf. I have rebased the branch.