User labeling and model training of fall detection model

[ivelin]

Is your feature request related to a problem? Please describe. Currently we use a simple heuristic that takes a before and after images from a fall sequence and compares the angle between the spinal vectors of the poses of the person in each image.

Sometimes this results in false positives , for example when someone is leaning to tie shoes or squatting.

Describe the solution you'd like We can use the simple heuristic model as a baseline and begin training a pose classification model to gradually reduce false positives. When a fall is detected with the current model, the user can be given an option to agree or disagree based on their own assessment of the before and after images. See attached diagram. The user can vote for each before and after image whether they agree or disagree with the estimation about the pose type (non-fallen person vs fallen person). In case they disagree, the use can choose one of two options, either pick the correct kind of pose (fallen vs non-fallen) or select Neither (not a person in the image or its hard to tell what the pose is - may only the head and shoulders are visible).

Once the end user provides feedback via PWA UI, the labeling data can be fed back to the Ambianic Edge for local training (potentially via TFLite Model Maker for image classification).

The custom pose classification model can be a type of image classification which is applied on black and white images that represent a person pose stick figure in white lines on a solid black background.

A further improvement on local model training, could be federated learning (with user opt-in) applied across all ambianic edge devices.

UPDATE, Aug 15, 2021: There is a good new tutorial on implementing a simple pose classification DNN that takes MoveNet key points as input. The example is for yoga poses. We can instead classify two labels: 'fallen person'/ 'not a fallen person'. We also need it done on TFLite Model Maker instead of TF/Keras. Currently TFLite Model Maker does not support classification based on sparse inputs such as pose keypoints, but we can try using the image classification API as described above. Draw pose keypoints as simple black and white stickfigure images and classify those with fall/no-fall labels.

Describe alternatives you've considered A more sophisticated approach will also improve missed fall detections. That requires further research since we want to keep user label effort to a minimum. Labeling missed detections means gathering user feedback on data samples which are not automatically detected. One potential approach is to occasionally ask user for feedback on fall detections with low confidence score.

Additional context

Flow diagram for labeling UX attached below.