Multiple outputs from Mapping Table

[Zejnilovic]

Background

Sometimes you might need 2 columns from the mapping table in your dataset.

Feature

Allow Mapping Table CR to have an output of more than one column

Example [Optional]

Mapping on ClientID column, wanting to take away name and surname from the MappingTable.

Proposed Solution [Optional]

  case class MappingConformanceRule(
                                     order: Int,
                                     controlCheckpoint: Boolean,
                                     mappingTable: String,
                                     mappingTableVersion: Int,
                                     attributeMappings: Map[String, String], //key = mapping table column, value = input df column
                                     targetAttribute: Array[String],  // <----
                                     outputColumn: Array[String], // <----
                                     isNullSafe: Boolean = false) extends ConformanceRule

Question

Would other rules benefit from this?

[AdrianOlosutean]

I would say this change should be universal for all rules, since the outputColumn is the same for all rules and it wouldn't necesarily make sense just for the mapping table in my opinion. I think this change would benefit all rules.

In case this idea doesn't seem feasible, I would suggest to have a new field such as additionalColumns:Map[String, String] just for MappingConformanceRule

[Zejnilovic]

I agree we make every CR output into list or array, but I would not "hard implement" it for other rules just yet. For other rules make it an array of 1.

This will also change our data model.

[AdrianOlosutean]

@benedeki @dk1844 Would you agree to proceed with @Zejnilovic 's proposal?