Zach

> * create a docu site for data quality? -> done > * check integration of original spark metrics. -> looks good @pgruetter : PR ok for you?

Yes, docu site is already existing, but description of expectations must be completed: http://smartdatalake.ch/docs/reference/dataQuality

I was able to run a model prepared by @raproth with SmartDataLake: ``` transformer.pythonCode = """ |import mlflow |model_uri = "file://c:/Entwicklung/test_mlmodel" |predict_udf = mlflow.pyfunc.spark_udf(session, model_uri) |from pyspark.sql.functions import * |inputDf.printSchema...

@pgruetter : can we exclude unit tests from the code to be inspected? see "sonar.test.exclusions" on https://docs.sonarqube.org/latest/project-administration/narrowing-the-focus/.

With this mechanism we should also collect metric numRecords for KafkaTopicDataObject and JdbcTableDataObject.

Interesting product: https://owl-analytics.com/

It's now possible to define Constraints on DataObjects that are validated for each row when writing. See also http://smartdatalake.ch/docs/reference/dataQuality#constraints

The solution would be to clone the Spark session (session.cloneSession()) in order to have isolated spark configuration for an action: https://stackoverflow.com/questions/47723761/how-many-sparksessions-can-a-single-application-have An Action with specific spark-options should clone the spark...

Point 3 & 4 are solved. But 1 and 2 remains, see WebserviceFileDataObject.

Hi @MarkusRothSBB, glad you found a workaround! I have the following improvement to the workaround: it should be .toInt instead of .toDouble at the line end: `case (d: DecimalType, _:...