jspsych
Losing around 10% of submissions
Thank you so much for this valuable tool. I've had a few issues recently where data from around 10% of my participants did not get saved properly to OSF despite them reporting no issues with doing the experiment. I haven't been able to identify a pattern for why this might be the case, but I am wondering if this is a known issue and/or others have experienced this?
I've experienced this occasionally but wasn't sure it was a DataPipe problem. Sounds like it could be. I'm gonna look into it!
Hi, I have experienced something similar. I lost about 50% of submissions in my last run of the experiment (that is, more people are indicated as having completed the study than there are datapoints). My session limit had not yet been reached but somehow no files (.csv format) were saved to osf beyond a particular amount. I don't think files are being overwritten because file names are unique subject id strings. Weirdly, when I run myself on the experiment as a test, a new file is created and the completed session count on datapipe goes up. I wonder if there is an internal cap to the number of synchronous sessions or something? I am not collecting much data at all (<100 subjects)
It seems like we might have this issue as well, any thought as to where it might come from?
I'm still stumped a bit by this. I thought it was folks closing the experiment early before the files are sent. @jadeddelta maybe we can look into this together?
when i did some test runs, data was still saved when I closed the experiment while the screen is displaying the saving data message. In the end, the data loss issue disappeared when i disabled data validation and session limit, but i still don't understand why, since i hadn't reached the session cap and neither did the missing data lack the required fields
Thanks @JosephOuta that's super helpful as a starting point for looking into it on our end.
In our case it happens with prolific participants, we had quite a couple of participants that received the payment code delivered at the very end of the experiment (after the data saving bits), and yet we don't have their data on OSF.
Given that it was quite large data files I thought maybe some internet instability would interrupt the participants' data transmission, but they would still move forward in the jspsych timeline (as afaik there is no callback that checks that the data has well been sent and received). So we tried additionally sending a stripped down version of the data, but the same thing happened (we didn't had any data for some participants despite them moving till the end of the experiment and receiving the prolific code)
We will try deactivating data validation & session limit and we'll see if it helps!
I work with @DominiqueMakowski and wanted to update you on this issue. We have two studies where data loss has occurred, ranging from 15% to 20% across two different platforms (Prolific and SONA). We only remarked it for the current Prolific study and went back and checked a similar SONA study to notice it happened too.
The main thing about these two paradigms is that they both collect webgazer eye tracking data, which might (for lack of better guess) be the culprit?
For the Prolific study, we attempted multiple solutions. First, we saved the data in two instances—one without the eye-tracking data and one with all of it—thinking that the issue might be related to eye-tracking data being too heavy and silently failing to transfer, which would mean that for these participants we would have only the "light" data and not the full one.
However, this did not resolve the problem (we don't collect any data for some participants), which might mean that the problem is not with the eyetracking data (again, it's just a guess).
We also updated the settings on DataPipe, following @JosephOuta's approach, but to no avail...
Is there a way for the datapipe plugin to perform some sort of check to make sure the data has been saved and received by OSF?
Is there a way for the datapipe plugin to perform some sort of check to make sure the data has been saved and received by OSF?
This is a great idea - I'll explore what is possible here.