Validation procedure

[tecamenz]

According to your notebook, you create your validation set based on a sample of 20% of the initial data set: df_ml_conso_for_model, df_ml_conso_validation = train_test_split(df_ml_conso, test_size=0.2, random_state=42)

If this is the case, you validate your final models on the same subjects you trained it on. While not exactly the same samples are used for validation, the samples come from the same subject and would therefore bias your results.

Validating your model on unseen subjects from the same dataset, I got a mean precision of 0.63 (SD=0.19), mean recall of 0.97 (SD=0.06) and mean F1 of 0.75 (SD=0.15). See also image below. The low precision indicates that a lot of windows will be marked good when they are not.