Uplift and Cross Validation

[shaddyab]

Should stratified cross-validation based on the Treatment vs Outcome 2x2 matrix split be used when performing a grid search to ensure that each fold follows the same distribution used in the overall data? If this is not the case, and cross validation is used for hyperparameter search, should we expect that the scores used for evaluating each fold, using qini coefficient as an example, indeed represent the qini coefficient for the overall training dataset? Putting it differently, an increase in number of folds may affect the stability of the uplift score in each fold.

Another closely related question, given the following two cross validation outputs, which one should we prefer? Higher mean score across folds, or uniform score across folds?

Number of cross folds: 4 Split_1_Score | split_2_Score | Split_3_Score | Split_4_Score | Mean_Score 0.4 0.9 -0.2 -0.3 | 0.2

Number of cross folds: 2 Split_1_Score | split_2_Score | Mean_Score 0.12 0.16 | 0.14