suggestion: split between train/test set, allow training and loading of imputation statistics models

[xuancong84]

In research, scientific integrity plays a very important part. One can publish very good papers by playing tricks between train and test set in order to get good results, but such results can never be applied in real life, because those tricks simply does not work in real-life applications.

Thank you very much for creating a wonderful framework for missing value imputation! However, your framework does not provide a way to apply imputation statistics trained on one dataset onto another dataset. I would greatly appreciate if you can make it.

For downward compatibility, you can create an optional kwarg called model for every function such as impy.mean, impy.mode, etc. When calling the function, by default model=None; if you pass model=True, the function will return a tuple consisting both the imputed data and the imputation statistics object; if you pass model=<imputation-statistics-object>, then the function will apply the trained imputation statistics to impute the data. In that way, all existing code will not be affected.

[Sandy4321]

will be great to do this

[Sandy4321]

seems to be scikit learn have this? https://scikit-learn.org/stable/modules/generated/sklearn.impute.KNNImputer.html#sklearn.impute.KNNImputer they do have transform(self, X) | Impute all missing values in X.

fit(self, X[, y])	Fit the imputer on X.
fit_transform(self, X[, y])	Fit to data, then transform it.
get_params(self[, deep])	Get parameters for this estimator.
set_params(self, **params)	Set the parameters of this estimator.
transform(self, X)	Impute all missing values in X.

[xuancong84]

Thanks @Sandy4321. I am aware of sklearn. Nevertheless, if you can make your code good enough, you can contribute your code into sklearn -:)