Filter pupil_width and other eye tracking columns similarly to pupil_area

[matchings]

Outlier removal and blink filtering was applied to pupil_area, eye_area, and cr_area in the eye_tracking attribute of the BehaviorOphysExperiment class, however a few additional columns were overlooked and should also have these corrections applied. Specifically, pupil_width, pupil_height, eye_width, eye_height, and potentially pupil_phi and eye_phi. If only one of these could be corrected, pupil_width (aka pupil diameter) would be the most important to do because it is most likely to be used for analyses.

I addition, it needs to be made very clear in the documentation which of these columns has had smoothing and blink correction applied. For example, if we do not apply blink correction to all of the _center_ columns, users will need to know to do that themselves.

We may also want to consider applying correction to all columns and having two tables, similar to running speed, one called eye_tracking where everything is corrected, and another called eye_tracking_raw where nothing is corrected.

List of all columns in eye_tracking:

for reference @DowntonCrabby @dougollerenshaw @yavorska-iryna

AllenSDK 2.10.1

[wbwakeman]

We could apply the likely blink filter to these other columns. But also the information is already there so that a user could easily apply the likely blink information to the other columns for themselves.

[djkapner]

For example: https://github.com/AllenInstitute/AllenSDK/blob/ea8cdc760210bec5ad01a3bd8fd586b6e2228080/allensdk/brain_observatory/behavior/session_apis/data_io/behavior_ophys_nwb_api.py#L272-L275

[matchings]

documenting that this has been an issue at SWDB and should be revisited after the course, along with other NWB related fixes.

the issue is not simply that likely blinks have not been excluded, but that the smoothing / filtering process to get rid of extreme outliers in the trace is not applied, making the pupil_width and other columns essentially useless. it is one thing to ask users to mask out likely blinks, but requiring users to come up with their own outlier detection filter is too much to ask (IMO). Especially when the result may be that pupil_area and pupil_width are filtered inconsistently with each other (one with our method, one with external users method).

in addition, the smoothing / filtering that is applied to pupil_area appears to be insufficient in many cases, as significant outliers and anomalies remain.

[aamster]

@matchings the last thing that you say is that

in addition, the smoothing / filtering that is applied to pupil_area appears to be insufficient in many cases, as significant outliers and anomalies remain.

This seems like a separate issue. My understanding of this ticket is that we would like to apply the same outlier removal and blink filtering that we currently apply to pupil_area to pupil_width, pupil_height, eye_width, eye_height, pupil_phi and eye_phi

If the current algorithm for outlier detection needs improvement, that seems like a separate issue. Is there a ticket for this?

[matchings]

@aamster yes i agree, improvement to the outlier detection algorithm should be a separate ticket. i have not made such a ticket, and i should do some additional characterization to clearly describe the issues with the current filtering prior to making a formal request. i mentioned it in this issue as a reminder to self and heads up for Pika, so i think it has achieved that purpose. But we can keep the issues separate.