Issue with DropColumnsTransformer when split is "chronological"

[ottobricks]

Since yesterday, I tried using FamilyRunner and it works past DropColumnsTransformer stage as long as I don't use "chronological" split method -- but fails in DataSplitUtility.split as reported here The error I get with FamilyRunner is different from the above. In my understanding, DropColumnsTransformer drops tunerTrainSplitChronologicalColumn despite the fact that I add it to fieldsToIgnoreInVector.

In my understanding, columns in fieldsToIgnoreInVector should be left untouched by all transformers, but it doesn't seem to be the case. It is possible to spot the problem with the debug flag. In my experiment, tunerTrainSplitChronologicalColumn -> "id_col", but it is not present in the step output dataset:

...
Output dataset schema: root
 |-- label_col: integer (nullable = true)
 |-- automl_internal_id: long (nullable = false)
 |-- features: vector (nullable = true)

=== End of class com.databricks.labs.automl.pipeline.DropColumnsTransformer Pipeline Stage log <==

I will look deeper into this and open a PR to fix it.