jsPsych
jsPsych copied to clipboard
jspsych
Use requestAnimationFrame when possible to pseudo-sync with display refresh.
requestAnimationFrame() is partially supported in the major browsers, and represents an improvement from setTimeout for measuring when stimuli are displayed.
The pluginAPI should get an additional method to display a stimulus for Xms. This method can fallback to setTimeout when needed, but allow requestAnimationFrame when possible.
The init() function should check browser support, and record which method is being used.
Hey, cool library! I just had an email discussion with Stian Reimers who had a recent BHS paper on timing accuracy using standard timestamps. He was kind enough to re-run his study using our setup (rAF, caching DOM selectors, using performance.now, minimising reflows). He found that rAF was much more consistent (almost four times lower SD 2.2 ms v. 9ms, basically it always rounds up the number of monitor refreshes while setTimeout can round down too). The mean duration was longer than intended, but I believe that was due to using 150 and 50ms durations, which aren't divisible by 16 (his screen refresh rate was 60Hz).
Until today I was not aware of a quick and easy method of determining the client's screen refresh rate. But it is possible by just requesting twenty animation frames and averaging the distance between them. I found someone who already did the legwork. See his fiddle http://jsfiddle.net/rBGPk/ or this variant that I made (it gives the minimum time distance between frames that your computer can do http://jsfiddle.net/bn8kbw3t/).
In our implementation we didn't sync to the refresh rate (or basically we decided we wanted a 500ms minimum). But if you add this to jsPsych you should probably first determine the refresh rate (the browser windows needs to be in focus for that) and leave it up to the user whether they want to hit their desired duration on average or as a minimum or as a maximum. And probably everybody would benefit from choosing presentation times that fit too a refresh rate of 60Hz (i.e. multiples of 1000/60).
If you want to have a look at our implementation (we rolled our own), it's on github too: https://github.com/rubenarslan/attentionbias
I can also mail you the examples that Stian used, if you're interested.
Thanks for the helpful info! I'd be happy to get some examples of how you and Stian have done this. When I ran the fiddle on my laptop, I got a 20ms interval. Does that mean that I would need to specify a presentation time that is a multiple of 20 to get as close to an exact display time as I can?
I think the main advantage of the rAF method is getting a good estimate of when a stimulus appears on the screen, for measuring response time.
This exact issue gave me a few headaches while working on the QRTEngine.
Yes you would need to specify a multiple of 20 to get the presentation time as close as possible. Actually, # Frames * 20 ms - 2 (or some other small fraction of the actual interval) to account for errors in rounding of the timestamp.
Next to that, take note that rAF is called somewhere in the interval between screen refreshes, and there is currently no support for retrieving the timestamp of the actual screen refresh. This can cause 'rounding down' of presentations by 1 frame (but no more). We mentioned this in our paper, but didn't go into an explanation why.
The following was taken from an internal discussion we held, after someone asked why we found that sometimes frames are 'dropped':
This is correct, some frames were indeed dropped during studies, leading to presentation times shorter than intended. This is a consequence of the fact that, while the rAF function notifies the QRTEngine between every refresh, this notification does not always take place at the same moment between refreshes. Hence, in rare occasions it can happen that two subsequent notifications (corresponding to two refreshes of 16ms. In such cases, the QRTEngine will incorrectly conclude that two frames have passed by. This happens only rarely and importantly, a maximum of one frame can be dropped. Accordingly, in the manuscript, we present the presentation timing accuracy as ± 1 frame deviation.
So, from the delta between subsequent rAF callbacks, you'll need to estimate the amount of frames that have passed. If refresh time is 20ms, it would be very likely you'd find a delta between 10 and 30 ms for one refresh. 10 ms if the refresh 1 callback happens late, and refresh 2 callback happens early. 30 ms if refresh 1 callback happens early, and refresh 2 callback happens late.
I hope this, unsolicited, advice will help you out a bit :) Good luck!
I was wondering about the time stamps on the rAF method. From what I can gather so far, it sounds like the rAF callback will trigger just after a display refresh, and the time stamp is when the rAF method triggers, so this might actually cause the response time measure to be overestimated by more than setTimeout, though the standard deviation will be lower.
But doesn't the callback passed to rAF get exactly that, i.e. a performance.now timestamp?
Cool that you two are apparently in touch. Stian referred me to both your papers! Maybe between the four of us we can put together some sort of best practice algorithm.
The timestamp that is passed back is created only slightly before the actual rAF is called, and I think it is non-standard (only firefox supports it, if I recall correctly). Let me check that!
Because of the high-priority of the callback, it will often be ran very close to the actual screen refresh, however this is no guarantee. And thus, because of these edge cases (where rAF can be called 1 ms before the next screen refresh), taking the current timestamp as the actual screen refresh time is very dangerous.
@ErwinHaasnoot I don't really get why you'd count frames at all? I mean frames can get dropped, that's sort of the point with rAF, it's sort of "the best you can get" but depending on what else is happening sometimes you don't get the best. I thought it'd be possible to usually get the best by instructing participants to close resource hogs and going full screen (our RT task in fact interrupted when you left fullscreen).
I don't think only FF supports the callback thing, at least the fiddle I showed relies on it and I ran it in Chrome. But it's true in our study I used Joe Lambert setTimeout drop-in replacement (requestTimeout) and it calls performance.now() in the callback, doesn't receive it as an argument.
Have you seen this Stackoverflow question I asked? I used the MozAfterPaint event to get a better grip on it. However, that's not turned on by default and only available in FF.
So: https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/RequestAnimationFrame/Overview.html#processingmodel
It says the following: Let time be the result of invoking the now method of the Performance interface within this context.
A context, in this context, is basically a function callback + invoking the correct scope. So performance.now is used only after the context has been created, which is slightly before the callback is called. Thus there is no guarantee that the timestamp is the actual timestamp of the screen refresh.
But the fiddle I showed gives me an average of 16.719 which is pretty close to the expected 1000/60 = 16.66666. I thought fewer rounding errors by using floats instead of integer milliseconds was one of the points of rAF. Maybe this is also a good query to some of the HTML5 bigwigs, I mean they clearly had something else in mind when they developed the spec, but maybe they can clear up some stuff.
I have a lot of ingrained assumptions when it comes to rAF from the many tests I performed. I think counting frames is necessary for some stuff (maybe its QRTEngine specific), but I can't recall exactly why.
Ah, cool that you dug this up. Actually I think I wrote a query to the webperf working group when I first happened upon this and found it somewhat wanting for reaction time studies, but can't find it now.
Do you use an initialiser to determine the screen refresh rate like the one I mentioned above @ErwinHaasnoot ? I didn't go that deeply into QRTEngine (don't have Qualtrics..) but didn't see any mention of it. Because it seems knowing that missing piece would help us get the lowest variability AND the most accurate presentation times.
Of course you need to be willing to adjust nominal presentation times slightly, but if you know what's going on, i.e. that you won't achieve nominal presentation times if you don't adjust, you should always be willing, right? That way you'd get less intertrial noise.
By the way do you think the time is already ripe to do some sort of absolute reaction time tasks? E.g. four choice RT or simple RT? We've been considering it, I thought we might be at the point where individual differences are bigger than device differences.
Also, has anyone done any blackbox measuring on mobile platforms?
This is all going a bit too fast :D I'm working on an example that shows why my previous post is important, even if you're not counting frames.
To answer your first question, the QRTEngine determines the screen refresh rate during the Init of any first trial in the block. I'll try and link to the specific code in a second. This is usually about 1000 ms, so generally enough samples to give a good estimate of the refresh rate. Ps: you can get a trial account for Qualtrics, but as you seem to be quite a competent programmer yourself, you would probably be best off sticking with jsPsych.
I didn't write the paper myself, and was quite busy the past year working on my Master's, so haven't really worked through the contents of the paper myself yet. If there's no mention of the refresh duration estimation, then that's probably my fault for failing to tell people about it :\
As for the exact location in the code where we check:
https://github.com/ErwinHaasnoot/QRTEngine/blob/master/lib/QRTEngine.js#L744
which are callback functions as well, and they're called in our draw loop:
https://github.com/ErwinHaasnoot/QRTEngine/blob/master/lib/QRTEngine.js#L1292
Conceptually, the issue of rounding down frames only happens when a stimulus needs to be displayed for more than 1 frame (ie, 2).
Consider this: 40 ms display of stimulus, 2 frames, as in my previous example.
The following flow happens: stimulus is queued, display should be synced with the screen refresh, so we queue our draw function
rAF callback - stimulus is displayed, onsettime is saved (current timestamp, best we can do), draw function queued with rAF rAF callback after 1 frame - timestamp should be onsettime + 20 ms, store this timestamp as duringtime, no 40 ms has passed yet, queue draw again rAF callback - onsettime + 40 ms should be <= to current timestamp, thus remove stimulus from display again. Store timestamp as offsettime
So we have three timestamps, onsettime, duringtime and offsettime, that should, relatively to some global onset time (page onset), be respectively 0 ms, 20 ms and 40ms in a perfect world, where rAF callbacks happen exactly upon the last screen refresh.
But we know this is not the case. the timestamp can be anywhere between previous and the next refresh (statistically, it's more often closer to the previous refresh). Let's see what happens if a rAF callback is dropped.
timestamp onset = 0 ms duringtime rAF callback - SKIPPED timestamp offset = 40 ms
Offset - onset = 40ms, thus 40 ms duration has passed, thus the stimulus should (correctly so, be hidden).
However, what if the onset timestamp is not 0 ms, but 1 ms? This results in offset - onset being 39 ms, below 40 ms, thus the stimulus should not be hidden, even though 2 frames have passed! However, 39 ms duration between stimulus onset and current timestamp is not unique for this situation, as the 39 ms duration can also occur if only 1 frame passes, and the current timestamp is actually 39 ms.
This requires thresholding based on knowledge about the statistics of rAF, a callback at 1 ms before the screen refresh is extremely unlikely, so we should count 39 ms as 2 frames having passed, and thus the stimulus should be hidden, rather than 1 frame having passed and we should keep displaying the stimulus.
However, although 39 ms is extremely unlikely, it is not impossible. So, in some cases, we will hide the stimulus, even though only one frame has passed, simply because the delta between two rAF callbacks is so high. You can see how lowering the threshold from 39 -> 38 -> 37 will cause dropping of a frame to happen more and more often and it's not all that clear what the threshold actually should be.
For the QRTEngine, we decided not to threshold, as we couldn't settle down on a proper value for this threshold. Because of this, sometimes, we display a stimulus too short, but it's probably the best we can do, as the actual value of the threshold will depend on the distribution of the timestamps across the refresh interval, and this distribution is likely to be different for different machines..
TL;DR, not knowing the timestamp of the actual screen refresh causes issues with displaying stimuli too shortly. In contrast to machine load, which can cause issues with displaying stimuli too long.
Ah cool! Well I'll happily slow down this conversation as it's dinnertime for me soon. Thanks for the details. Really interesting. Looks like your rationale is very sound and you did a lot of research into the nitty-gritty. Having an idea of the number of frames seems important, so that you can guess what happened. I wouldn't have thought that there's so much variation to what timestamp you end up with. Seems like a somewhat bad implementation actually? Do you reckon there's any benefit to be had by doing dual timing with setTimeout or so?
Has either of you looked into minimising reflows and repaints when drawing (e.g. using css visibility instead of display, blocking animations)? I think this made some difference for me, but I didn't have a blackbox to test, I just MozAfterPaint.
Yeah, I won't be using Qualtrics, actually I made a open source alternative, so I wouldn't lead by a good example if I went to Qualtrics ;-). I haven't used jsPsych either, I just wrote my own since I couldn't find your software (or maybe they didn't exist ~two years ago when I implemented it). I shouldn't have, caught some errors pretty late in the data collection, but oh well, so you learn.
Hey but if either of you want to interface your software with formr, I'd love to support that. I think we're already close to the cutting edge when it comes to survey research and experience sampling, but since I work in a personality department the reaction time stuff got less love. And at least when working with formr you wouldn't have to do lots of workarounds to get a proper data storage for your RTs :smile:
To come back to this topic and answer some of the remaining questions you had, Ruben.
First off, I actually ran across Stian Reimer's paper in BRM, and showed it to my colleagues, but I'm sad to say, I dismissed it out of hand, because rAF wasn't used in the JavaScript experiment. (I'm sort of a fanboy..) The methods seem to be sound though, so I'd love to see a properly done comparison between rAF and Java/Flash-based experimentation.
-- Have you seen this Stackoverflow question I asked? I used the MozAfterPaint event to get a better grip on it. However, that's not turned on by default and only available in FF.--
I'm not sure using 'mozAfterPaint' is the correct approach to this issue. You're still reliant on setTimeout to "deliver" you to somewhere around the correct time, which it very likely doesn't (the callback is of medium priority, and thus can get significantly delayed, compared to rAF). Looping based on 'mozAfterPaint' would then only result in a bad rAF-like function, because it is a normal event that enjoys no special priority privileges as far as I know..
-- Of course you need to be willing to adjust nominal presentation times slightly, but if you know what's going on, i.e. that you won't achieve nominal presentation times if you don't adjust, you should always be willing, right? That way you'd get less intertrial noise. --
I'm not sure what you mean by nominal presentation times (I'm not too well-versed in cognitive science terminology). It sounds interesting, so please do explain
-- By the way do you think the time is already ripe to do some sort of absolute reaction time tasks? E.g. four choice RT or simple RT? We've been considering it, I thought we might be at the point where individual differences are bigger than device differences. --
I do think the time is ripe, mostly because variance due to differences in machines can be overcome by increasing your amount of participants, usually a trivial thing to do in online studies. We have been able to get effects of about 10 ms significant (the negative compatibility effect in our Masked Priming study), while not even having that many participants (80 if I recall correctly, which is a very small amount for an online study). Henk van Steenbergen has been running more experiments, which I don't know the details about, but was assured that they show very powerful effects as well.
-- Has either of you looked into minimising reflows and repaints when drawing (e.g. using css visibility instead of display, blocking animations)? I think this made some difference for me, but I didn't have a blackbox to test, I just MozAfterPaint. --
It should make a difference, because using CSS visibility doesn't cause the page to be re-rendered (I'm not sure about the terminology here), ie, sizes and positioning of elements do not have to be recalculated, visibility: hidden elements are simply transparent. This behaviour is very useful if you build your experiment from the ground up, but sadly I'm stuck with what Qualtrics provides me, and thus I can't really make use of it. I'm just glad that most experiments tend to be very simple in what needs to be painted.
-- Hey but if either of you want to interface your software with formr, I'd love to support that. I think we're already close to the cutting edge when it comes to survey research and experience sampling, but since I work in a personality department the reaction time stuff got less love. And at least when working with formr you wouldn't have to do lots of workarounds to get a proper data storage for your RTs --
Sounds cool! The future is in open source frameworks that allow you to do both normal questionnaires and also RT experiments easily. Commercial opportunities would then come from hosting of the framework and customizing it. The QRTEngine is a pile of bandaids to get RT working in Qualtrics, and it doesn't look like Qualtrics is willing to provide much support for it (to my extreme frustration) . So that means.. well, probably shouldn't go there.
Anyway, I sadly don't have time to work on this stuff, as my time at university is coming to an end..
I think Stian plans to post the results from the rAF comparison on his website and he's the one who clued me in to your papers, so I think he's open to being convinced of rAF.
The Stackoverflow question is a bit misleading. At the time I asked the question I was using setTimeout, but if you look at the answer, I compare inline execution timing and MozAfterPaint timing with setTimeout to with rAF. And what I saw was that the Paint-to-Paint times scattered more realistically and didn't have these far outliers (which I guess are due to setTimeout being medium-priority). Also, execution time and paint-to-paint correlated in this condition and I was able to enforce a minimum display duration. All I wanted to point is that it appeared possible to set the MozAfterPaint event handler inside the rAF callback, before the painting was done. But maybe I was misled, as I didn't have any outside validation.
"nominal" presentation times isn't special jargon, maybe I used the wrong word. I just meant: if people want to achieve a duration of 50ms, they would usually specify that. But if you can show them that this doesn't make sense (most common screen refresh rate = 60Hz), so you achieve 48 sometimes and 64 most other times (inconsistently, driving up random error) and get more between participant error (someone with a refresh rate of 50Hz would average a different presentation time), then a good researcher should want to get the right actual time instead of the right "nominal"/declared time.
Re absolute RT: This isn't exactly what I meant, we were able to get a significant difference of 13ms in an RCT training task as well, but you need a lot less power for a between group test and if you randomise confounds on the participant level just add to the error. In the four choice reaction time task we would use this as a measure of processing speed and correlate it with other measures of e.g. verbal IQ. So device differences and between-participant differences would be confounded. So we wouldn't want to conclude that poor users have lower processing speed because they use older devices. However, I find the prospect of controlling for the screen refresh rate appealing, since this seems to be one of the main remaining error factors. I only recently realised that it is measurable, before we only collected browser, OS and resolution data. One hidden confounding variable that 'd be left would probably be keyboard scanning speed?
Re reflows v. repaint: yeah a reflow is a re-rendering which can be minimised by using visibility instead of display and absolute positioning etc. Though you sometimes have to do some manual tracing in Chrome etc. yourself because it's sometimes surprising what causes and doesn't cause a reflow.
I have to say for a pile of bandaids you put a lot of thought in to QRTE. Too bad Qualtrics isn't interested in supporting it. Do I understand you correctly that there is no plan to maintain the software? That's unfortunate! If you want to exchange experiences about maintaining open source scientific software, please email me, I have a feeling we're kind of hi-jacking these issue comments :-)
I'd love to know as soon as Stian posts his rAF findings on the website.
I understand now, I didn't work through the actual chosen answer. I'm still a bit confused as to the meaning of 'inline execution' timing. Is it an alternative way of measuring the duration of the display (otherwise looking at scatterplots would be sort of meaningless). See how they correlate? Next to that, if I understand correctly, in your rAF solution, you set the stimulus to visible, use that rAF requestTimeout thing to timeout for 500 ms, and then after 500 ms has passed call your function to hide the stimulus again. This means the display of the stimulus is not synced to re-paints, thus you introduce unnecessary variance in your solution.
50 ms is 3 frames on a 60 Hz screen, so not a good example ;) but I understand what you mean now. Offline software, if ran on PCs with CRT displays, was able to adjust the refresh rate the screen was running at, which would allow them to work reasonably well with any intended duration by simply adjusting the refresh rate so that the intended duration would be a multiple of the new refresh rate. This is no longer possible, and it begs the question whether you should start filtering based on refresh rates (anything but 60 Hz not being allowed).. Also, if you define, for example, 52 ms duration stimulus, on a 60Hz display, this should always be assumed to result in 66(.6666...) ms displays of the stimulus. 50 ms display could happen due to resolution issues of the timestamp, or what I previously mentioned with thresholding issues, but these are exceptions and shouldn't be very common.
Lets continue through e-mail yes, I currently don't have time to finish this comment and I don't mean to hijack anything ;) erwinhaasnoot[at]gmail.com!
Loving this conversation! I'm the chap behind www.xperiment.mobi (flash based ;) . Writing a review article about online research currently. Erwin, liked your blog about differences between QRTEngine and JSPsych link. Keen to quiz you before submitting! :)
Hi Andy,
Sure! You can reach me at the e-mail mentioned in my previous post :)
On Thu, Jan 29, 2015 at 2:30 PM, Andy Woods [email protected] wrote:
Loving this conversation! I'm the chap behind www.xperiment.mobi (flash based ;) . Writing a review article about online research currently. Erwin, liked your blog about differences between QRTEngine and JSPsych link http://www.qrtengine.com/qrtengine/comparing-qrtengine-and-jspsych/. Keen to quiz you before submitting! :)
— Reply to this email directly or view it on GitHub https://github.com/jodeleeuw/jsPsych/issues/75#issuecomment-72023513.
Hey @jodeleeuw, maybe this will become easier with JQuery 3 which now defaults to rAF for animations. Though you would probably still need a frame counter like we discussed above.
Thanks for the heads up! I hadn't seen the announcement. This is still on my list of things to tackle, but it's obviously a major change and one that I haven't yet found the time for.
Anyone know if the rAF test results have been published yet?
I don't think Stian Reimers put it on his website yet, at least I can't find it there. But I offered to forward you the report a while back, I will send it now. Of course switching to jQuery 3 might be a major change, I cited this more to show it's becoming established among animators (who've probably worried about dropped frames more than we ever will) :-).
@rubenarslan @ErwinHaasnoot Have you thought more about this in the past 18 months? The idea of a best practices algorithm has come up again in the context of another discussion and I'm curious if anyone has improved the state of the art.
I saw mention jquery3. Please be aware that eg https://greensock.com/get-started-js are rather critical of it and offer a nice alternative (not sure it's opensource).
I haven't kept up to date with the latest changes, sadly. Qualtrics broke the qrtengine a while ago, so haven't been keeping myself up to date. A quick survey of the available popular literature tells me that not much has changed. I still stand behind the QRTEngine method of using rAF (frame counting etc.) and I would wager distilling that algorithm would give you the best performance, even though I have no numbers to back it up.
I agree with @ErwinHaasnoot, switching to rAF using Erwin's method still seems like the best approach. I think it would greatly improve jsPsych and I think it's not as much of a change as it might seem. Stian Reimers recently wrote a new OA paper that includes test with rAF. I'm only judging from the fact that I switched a self-made script from setTimeout to rAF once and it wasn't that hard, Erwin's method is a bit more sophisticated though.
Too bad Qualtrics broke the QRTengine! Next time, go full open-source, use formr as a base ;-) Maybe you, @ErwinHaasnoot, could do a pull request for rAF, Josh and me do code review, and Stian could test the improvement with his blackbox? That would be open science at its best :-)
I've been thinking about writing some functions into the API for stimulus display, which gives me a chance to think about how rAF and the jsPsych API can play nicely together. But the first step is probably to try and settle the best-practices algorithm. Are either of you interested in collaboratively putting together a set of test cases? I've got my own BlackBox now, so we can test with that.
I think @ErwinHaasnoot mapped out very nicely what the optimal flow is to time animations and measure response times (in this thread) and in the QRTengine code, but maybe he could summarise it once more. I would add that if you're controlling stimulus flow, you should do work to minimise reflows and repaints, maybe by pre-rendering "slides" and just turning visibility on/off (instead of display). I'll gladly do some code review, but I haven't worked with reaction time stuff for a while now and I've never used jsPsych.