cloudflow icon indicating copy to clipboard operation
cloudflow copied to clipboard

Default parameters in Spark Streamlets are not processed correctly in tests

Open michaelpnash opened this issue 4 years ago • 0 comments

Default parameters of a Spark Streamlet are not correctly processed, at least within the context of a unit test.

e.g.:

"use the default value of a config parameter if the config parameter is not set" in {
      val configTestKit = SparkStreamletTestkit(session)
      // setup inlet tap on inlet port
      val in: SparkInletTap[Data] = configTestKit.inletAsTap[Data](instance.in)
      // setup outlet tap on outlet port
      val out: SparkOutletTap[Simple] = configTestKit.outletAsTap[Simple](instance.out)
      // send to inlet tap
      in.addData(sampleData)
      // run the stream
      configTestKit.run(instance, in, out, 2.seconds)
      // get data from outlet tap
      val results = out.asCollection(session)
      // assert
      results mustBe Array.fill(2)(Simple("initial"))
    }

See also: SparkStreamletSpec.scala

[info] SparkStreamletSpec
[info] - should use the default value of a config parameter if the config parameter is not set *** FAILED ***
[info]   com.typesafe.config.ConfigException$Missing: empty config: No configuration setting found for key 'name-filter-value'
[info]   at com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:156)
[info]   at com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:174)
[info]   at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:188)
[info]   at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:193)
[info]   at com.typesafe.config.impl.SimpleConfig.getString(SimpleConfig.java:250)
[info]   at cloudflow.spark.SparkStreamletSpec$MySparkProcessor$1$$anon$1.<init>(SparkStreamletSpec.scala:45)
[info]   at cloudflow.spark.SparkStreamletSpec$MySparkProcessor$1.createLogic(SparkStreamletSpec.scala:44)
[info]   at cloudflow.spark.SparkStreamlet.run(SparkStreamlet.scala:89)
[info]   at cloudflow.spark.SparkStreamlet.run$(SparkStreamlet.scala:81)
[info]   at cloudflow.spark.SparkStreamletSpec$MySparkProcessor$1.run(SparkStreamletSpec.scala:34)
[info]   ...

michaelpnash avatar Apr 23 '20 18:04 michaelpnash