skrub
skrub copied to clipboard
skrub-data
Adding a transformer to sessionize a table
Problem Description
In a table representing events with timestamps, it is often useful to detect sessions -- groups of events that occurred close to each other.
For example, a few visits to the hospital a few days apart are likely to be exams and follow-ups related to the same medical event. Or we may want to detect when a user walks away from an app, website, service etc and later comes back to it. Then features like number of sessions, session length etc can be extracted.
Examples from Ibis, polars and SODA using logs from a video game:
https://ibis-project.org/how-to/timeseries/sessionize https://www.pola.rs/posts/the-expressions-api-in-polars-is-amazing/ https://github.com/soda-inria/survival-analysis-benchmark/blob/main/notebooks/tutorial_part_2.py
Would it be in the scope of skrub to have a transformer that adds a "session id" column to a dataframe, and maybe extracts some features such as session length?
Feature Description
transform a table by detecting sessions and adding related information
Alternative Solutions
No response
Additional Context
No response
I wouldn't mind working on this if folks agree. This session stuff is the the prime villain in any story about data leakage in ML pipelines.
I wouldn't mind working on this if folks agree. This session stuff is the the prime villain in any story about data leakage in ML pipelines.
Awesome.
However, I would delay working on this, as with @JeromeDockes we are in the middle of a huge refactor of skrub.
We need to put notes of this discussion on the skrub repo
@koaning part of the changes Gaël mentions is being worked on in #877 Feel free to provide feedback and advice!
One observation since responding to this all those months ago ...
Do we generally want to add sessions to the dataset before calling train/test split? I think we might! The session may be what we need in order to split events into groups which define the train/test splits. If that's the case ... do we want to have a pipeline that runs before we split train/test?
do we want to have a pipeline that runs before we split train/test?
At some point we will have to address this, and I think that it means that at some point the "Recipe" should know how to split data.
In the short term, I would rather not move forward with showing explicitly a pipeline before the split, as I worry about leaks.