cloudflow
cloudflow copied to clipboard
Default parameters in Spark Streamlets are not processed correctly in tests
Default parameters of a Spark Streamlet are not correctly processed, at least within the context of a unit test.
e.g.:
"use the default value of a config parameter if the config parameter is not set" in {
val configTestKit = SparkStreamletTestkit(session)
// setup inlet tap on inlet port
val in: SparkInletTap[Data] = configTestKit.inletAsTap[Data](instance.in)
// setup outlet tap on outlet port
val out: SparkOutletTap[Simple] = configTestKit.outletAsTap[Simple](instance.out)
// send to inlet tap
in.addData(sampleData)
// run the stream
configTestKit.run(instance, in, out, 2.seconds)
// get data from outlet tap
val results = out.asCollection(session)
// assert
results mustBe Array.fill(2)(Simple("initial"))
}
See also: SparkStreamletSpec.scala
[info] SparkStreamletSpec
[info] - should use the default value of a config parameter if the config parameter is not set *** FAILED ***
[info] com.typesafe.config.ConfigException$Missing: empty config: No configuration setting found for key 'name-filter-value'
[info] at com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:156)
[info] at com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:174)
[info] at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:188)
[info] at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:193)
[info] at com.typesafe.config.impl.SimpleConfig.getString(SimpleConfig.java:250)
[info] at cloudflow.spark.SparkStreamletSpec$MySparkProcessor$1$$anon$1.<init>(SparkStreamletSpec.scala:45)
[info] at cloudflow.spark.SparkStreamletSpec$MySparkProcessor$1.createLogic(SparkStreamletSpec.scala:44)
[info] at cloudflow.spark.SparkStreamlet.run(SparkStreamlet.scala:89)
[info] at cloudflow.spark.SparkStreamlet.run$(SparkStreamlet.scala:81)
[info] at cloudflow.spark.SparkStreamletSpec$MySparkProcessor$1.run(SparkStreamletSpec.scala:34)
[info] ...