Add fingerprint stimulus to stim table

[wbwakeman]

BehaviorOphysSession.stimulus_presentations should also include rows for the fingerprint_stimulus that is shown at the end of each behavior session

[matchings]

There is a 30 second movie clip that is repeated 10 times at the end of each Visual Behavior ophys session. Users need to know the timing of this movie so that they can analyze activity during that period, and know when the preceding gray screen period ends. Ideally, the movie frame displayed for each stimulus frame would be provided, or at minimum the start and end time of each repeat of the movie should be provided. This information should be buried in the pkl file somewhere.

[aamster]

@matchings could you please help me to find the relevant data in the pkl file?

[matchings]

@aamster Here is a gist where i inspected the pkl file to figure out where the relevant info is. Hopefully it all makes sense. It took me a while to figure it all out since there are a lot of different pieces of information that are not well named or documented....

https://gist.github.com/matchings/dc13a5ea23ca19d229b3bd47abd83365

[aamster]

I'm unsure about a few things on how the fingerprint movie should be represented in the table. I'll go through each field in the stimulus presentations table

duration: I take this to be 1/60. Or is it 1/30?

start_frame: I don't know what is meant by "frame". I would expect it to start on frame 0. But it starts on frame 234034. What is the frame relative to?

end_frame: Is this just the frame after the current frame? Ie start 234034, end 234035 ? Should the subsequent row start_frame be the same as the current end_frame? Ie start 234034, end 234035, followed by start 234035, end 234036 ?

start_time: I'm not sure what this start_time is relative to. The start of the session? The start of the fingerprint movie? If start of the fingerprint movie, then it should be 3000, due to the 5 minutes of gray screen. If start of the session, then I think it should be the last end_time in the stimulus presentations table + 3000.

end_time: start_time + duration ? Should there be a gap between start_time of subsequent rows? or the next start_time should be the same as the previous end_time? I.e start 3000, end 3000.0167, followed by start 3000.0167, end 3000.033 ?

image_index: I think this should be nan

image_name: I think this should be nan

image_set: either "fingerprint_stimulus", something else or nan?

is_change: False?

omitted: always False?

All of this assumes that each frame of the fingerprint movie should be a single row in the stimulus presentations table. Or should an entire playing of the movie be a single row?

[aamster]

@matchings please see above. I can try to clarify my questions if they aren't clear.

[matchings]

@aamster sorry for the slow response. i am also unsure about some of these things. i think it could be worth considering 1) how the Visual Coding part of the SDK (which is where these movies original come from) represent this information in their stimulus tables, and 2) whether the fingerprint stimulus table should be separate from the main behavior stimulus table, and/or whether it should be labeled as a unique "stimulus block" as i believe the different VBN stimuli (behavior vs. receptive field mapping vs passive) are differentiated.

ill try to answer a few of your questions that i think i know the answer to, but i am not really positive about any of this because i haven't worked with the movie stimulus or the pkl file all that much myself.

duration: I take this to be 1/60. Or is it 1/30?

if we consider one frame of the fingerprint movie to be one stimulus presentation, and each movie frame is present for 2 stimulus frames (i.e. is on the monitor for two frames at 60Hz frame rate), then i think the stimulus duration would be 1/30.

start_frame: I don't know what is meant by "frame". I would expect it to start on frame 0. But it starts on frame 234034. What is the frame relative to?

i think this is the stimulus display frame index (monitor display at 60Hz) that a given movie frame (one frame of the 30 second natural movie clip) was displayed on. it should be relative to the very first stimulus frame of the entire session, which is 0. Then there is 60 mins behavior + 10 mins gray screen, at 60Hz display rate, so on the order of ~250k frames before the fingerprint starts makes sense to me.

end_frame: Is this just the frame after the current frame? Ie start 234034, end 234035 ? Should the subsequent row start_frame be the same as the current end_frame? Ie start 234034, end 234035, followed by start 234035, end 234036 ?

i think each movie frame is played for two stimulus frames. so i would expect to have one movie frame start at stimulus frame 234034 and end at stimulus frame 234035, then the next movie frame would start at 234036 and end at 234037. But we should check how visual coding does this.

start_time: I'm not sure what this start_time is relative to. The start of the session? The start of the fingerprint movie? If start of the fingerprint movie, then it should be 3000, due to the 5 minutes of gray screen. If start of the session, then I think it should be the last end_time in the stimulus presentations table + 3000.

end_time: start_time + duration ? Should there be a gap between start_time of subsequent rows? or the next start_time should be the same as the previous end_time? I.e start 3000, end 3000.0167, followed by start 3000.0167, end 3000.033 ?

image_index: I think this should be nan

we need to represent the frame index within the 30 second movie clip. whether we put this in "image_index" or another column called "movie_stimulus_index", i am not sure. As i said above, we should try to use similar nomenclature as the visual coding stimulus table if possible. if that's not possible for whatever reason, then using this existing column of "image_index" could be ok, as long as the movie stimulus is distinguished from the behavior stimulus by an additional column called "stimulus_block" (or just give the fingerprint movie its own stimulus table)

image_name: I think this should be nan

Same answer as above

image_set: either "fingerprint_stimulus", something else or nan?

i think it should be named consistently with the visual coding movie stimulus, which i think is something like "natural_movie_one"

is_change: False?

correct

omitted: always False?

correct.

All of this assumes that each frame of the fingerprint movie should be a single row in the stimulus presentations table. Or should an entire playing of the movie be a single row?

Again, sorry to have minimal answers here, but we should be consistent with what visual coding did. Let me try to find one of their tutorials that loads the natural movie clip stimulus table (aka "fingerprint stimulus") so we can see how it is formatted.

[matchings]

@aamster This notebook goes through how to load the various Visual Coding stimulus tables including "natural_movie_one" which is the same thing as the fingerprint stimulus. it doesnt display the stimulus table, which should include start_frame, end_frame, etc., but at least its a starting point for comparison

@mabuice probably has some helpful information here as well given that he designed that stimulus and the original SDK structures for it

[aamster]

A few observations from looking at the visual coding stimulus table for natural_movie_one:

• start_time and end_time are not included in the visual coding table
• There is a column called "repeat" to indicate how many times the movie has been played. We can add this and set it to nan for the rows for images.
• There is a column called "frame" to indicate the local frame index. We can add a column with the same name (or "movie_frame_index" which I think is more descriptive) and set it to nan for all rows for images, which already have a value for image_index. The value of image_index for the movie stimulus can be nan.
• For repeat 0, the values for the columns start and end is the same (I think this is a bug), but for repeats 1 - 9, there is a difference of 1 such as start 71354, end 71355, followed by start 71355, end 71356. That seems to indicate that each frame is just displayed for a single stimulus frame, unless it is different for vbo? For VBO, sweep_frames value of the pkl file stores it as [(0, 1), (2, 3) ...]. This leads me to believe that each frame lasts for 2 stimulus frames, but that's not how visual coding presented it.

To be consistent with VBN, I think we should add "stimulus_block" and "stimulus_name" columns to differentiate between behavior stimuli and the movie stimulus. The stimulus block in vbn is just an integer, and they include a "stimulus_name" column with the name of the block.

I also think we need a stimulus block for the gray screen? VBN calls this block "spontaneous" and records it as a single row in the stimulus presentations table.

[aamster]

@matchings can you please test the implementation on the ticket/1190 branch and let me know your thoughts?

i.e using

from allensdk.brain_observatory.behavior.behavior_session import \
    BehaviorSession
sess = BehaviorSession.from_lims(behavior_session_id=1192665673)
stimulus_presentations_table = sess.stimulus_presentations_table

[matchings]

@aamster very sorry for delayed response on this. I will test it out and let you know.

[matchings]

@aamster all the

A few observations from looking at the visual coding stimulus table for natural_movie_one:

* start_time and end_time are not included in the visual coding table

* There is a column called "repeat" to indicate how many times the movie has been played. We can add this and set it to nan for the rows for images.

* There is a column called "frame" to indicate the local frame index. We can add a column with the same name (or "movie_frame_index" which I think is more descriptive) and set it to nan for all rows for images, which already have a value for image_index. The value of image_index for the movie stimulus can be nan.

* For repeat 0, the values for  the columns `start` and `end` is the same (I think this is a bug), but for repeats 1 - 9, there is a difference of 1 such as start 71354, end 71355, followed by start 71355, end 71356. That seems to indicate that each frame is just displayed  for a single stimulus frame, unless it is different for vbo? For VBO, `sweep_frames` value of the pkl file stores it as [(0, 1), (2, 3) ...]. This leads me to believe that each frame lasts for 2 stimulus frames, but that's not how visual coding presented it.

To be consistent with VBN, I think we should add "stimulus_block" and "stimulus_name" columns to differentiate between behavior stimuli and the movie stimulus. The stimulus block in vbn is just an integer, and they include a "stimulus_name" column with the name of the block.

I also think we need a stimulus block for the gray screen? VBN calls this block "spontaneous" and records it as a single row in the stimulus presentations table.

Everything you said here makes sense and i agree, except for the lack of inclusion of start_time and end_time - while visual coding may not have had this, we do want these for VBO

[aamster]

@matchings I've made some changes in case you have pulled the code already.

I interpreted the sweep_frames which are nonoverlapping, i.e [(0, 1), (2, 3), ...] to mean that the frames are nonoverlapping. In other words, these indices are inclusive and 2 different movie frames can't appear on the same stimulus frame. So I updated the start/end frames to be nonoverlapping instead of overlapping.

I also updated the start/end times to use stimulus timestamps instead, which incorporate monitor delay. There are 36000 values of the stimulus timestamps after the last behavior stimulus frame, which I'm assuming correspond to the gray screen + fingerprint movie.

[matchings]

Hey @aamster unfortunately im not going to have time to look through this in the next few days, but @nataliaorlova @mattjdavis and perhaps @DowntonCrabby have some interest and investment in using the fingerprint stimulus, so I'd like to nominate the to take a look at the new stimulus table you made and provide any feedback they have about interpretability, utility for doing specific analyses, and consistency in formatting with other types of stimulus tables.

[aamster]

It's not a new stimulus table but has been added to the existing one. To access it, use branch ticket/1190 and i.e.

from allensdk.brain_observatory.behavior.behavior_session import \
    BehaviorSession
sess = BehaviorSession.from_lims(behavior_session_id=1192665673)
stimulus_presentations_table = sess.stimulus_presentations_table

It's best to get this reviewed quickly so there isn't stale code that causes merge conflicts. It shouldn't take too long to review

[mattjdavis]

@matchings @aamster Looks great to me, interpretable and good utility (as somebody just getting used to the stim table). It is clear how to get the various stim blocks (behavior, spontaneous, natural movie). I can't speak to consistency in formatting with other stimulus tables.

If I were to design the tables from the jump, I would rename repeat --> movie_repeat for clarity. But again seems consistency with other tables supersedes that, so I am cool with the current name.

Sidenote 1, the last stim of the movie is 12 seconds, perhaps a hardware/rig quirk that we can discard at analysis or ask MPE to look into.

Sidenote 2, last line in example above should be stimulus_presentations_table = sess.stimulus_presentations