[WIP] Interactive visualization of detectors

[ljchang]

In this PR, i have rewritten the plot_detections() method of Fex objects to use plotly backend instead of matplotlib. There are now two new ways to create interactive visualizations of detectors using plotly. Initial discussion in issue #177 and addresses enhancement in #46.

plot_singleframe_detections() - plots a single image frame from image or video and allows users to interactively toggle individual detectors. Unfortunately, users can currently only toggle a single detector, I have not yet been able to figure out how to elegantly allow the indepedent toggling of detectors without creating separate buttons to toggle each on and off.

plot_multipleframes_detections() - plots a series of image frames as a plotly animation and allows users to interactively slide through detection results for individual frames. The specific type of detector to plot must be prespecified as a boolean when calling the function.

Ideally, we would be able to combine these two different types of plots into a single plot, but I have not yet been able to get it to reliably work yet. While plotly offers a lot of flexibility, the two different types of interactivity create lots of conflicts with each other. For now, the updated plot_detections() method automatically detects which type of plot to use.

Here are the tasks to complete:

• [x] add facebox visualization
• [x] add landmark visualization
• [x] add pose visualization
• [x] add au visualization
• [x] add emotion visualization
• [x] add positions for emotion annotations
• [x] add slider to scroll through frames
• [x] add buttons to toggle landmarks, faceboxes, au maps
• [x] turn into plot_detections() method
• [ ] streamline muscle au dictionary
• [ ] add examples for saving plots and videos
• [ ] add tooltips to visualize detector output for each face
• [ ] allow multiple detectors to be toggled in plot_singleframe_detections()
• [ ] add adjustment buttons for playback speed in plot_multipleframes_detections
• [ ] ideally, both types of interactivity would be combined into a single plot
• [ ] add tests for figures. I'm still not sure what the best way to do this is.
• [ ] add new function to plot faces from AUs (the other main way to generate plots in py-feat)
• [x] add new function to create live plots

I still need to streamline the AU dictionaries and confirm the drawing locations are accurate that @skbyrne started. Will need one of my AU anatomy books for this.

I will need some help testing this to make sure everything is working reliably and also fixing some of the idiosyncratic spacing issues. I also still need to write tests, but am not sure what the best practices are for writing plot tests in python.

@ejolly, when you have some time it would be great to get your feedback on this.

[ljchang]

@ejolly , I've added code to create interactive plots with real-time py-feat detection on live captured data suggested in issue #168. This is fairly complicated to get working in a jupyter notebook and I have no doubt that there will be lots of bugs to iron out as we test it on different systems and notebook environments.

Here is example code for how to run it.

from feat.plotting import pyfeat_live_demo

pyfeat_live_demo(save_fex_file = '/Users/lukechang/Downloads/pyfeat_live_fex.csv', max_frames=50)

The frame rate is fairly slow due to the current speed of py-feat, but that can only get better.

[ljchang]

Discovered another annoying bug that is an interaction between versions of plotly and ipywidgets. See https://github.com/plotly/plotly.py/issues/3686 for more details. Pinning ipywidgets to 7.8.0 for now.

[ljchang]

I'm also finding that the main figure is not rendering correctly using jupyterlab or vscode notebook extension. This needs to be investigated deeper. I have a feeling it is related to the incompatibility of plotly FigureWidgets with the newer versions of ipywidgets. I suspect that each different notebook version has it's own widget package that needs to be pinned, which is really annoying.

Basically, I'm running into an issue that the FigureWidget object does not correctly work with ipython display in ipywidgets >=8

AttributeError: type object 'DOMWidget' has no attribute '_ipython_display_'

Not sure why plotly hasn't fixed this issue yet.

[ljchang]

There is an issue with plotting very large photos that we will need to adjust with some autoscaling.