Asynchronous loading of images during experiment runtime

[nikbpetrov]

Problem statement:

I have loads of images. I want to preload those images in the background during the first N trials, until I hit a spot where the user cannot continue until images are loaded.

What I have tried

The jsPsych preload functions and plugin did not seem to cut it for asynchronous loading so a simple use of Javascript should do the case -- but I cannot seem to get it to work with jsPsych. The idea, I think, should be simple: initialize the experiment with at least the first N trials (but see below for variations) and at the very first trial, send asynchronous requests to start loading images. Then at whatever point, ask for the Promises of the requests to be resolved and only then continue. To that effect, here's some dummy code:

var timeline = []

// an array holding loads of images
var images = ['img_1.jpg', 'img_2.jpg']
// when we fetch the images, we want to store an array of promises here
var PROMISES = []

var start_experiment = {
	type: 'html-keyboard-response',
	stimulus: 'Press any key to continue',
	on_load: function() {
		// here, on the first trial, we request the images and keep the promises in the PROMISES array to use later when we want to stop
		for (img of images) {
			promise = new Promise((resolve, reject) => {
				$.get(img)
					.done(() => resolve(1))
					.fail(() => resolve(0))
			})
			PROMISES.push(promise)
		}
	}
}
timeline.push(start_experiment)

var trial_during_which_media_needs_to_be_loaded_1 = {
	type: 'html-keyboard-response',
	stimulus: 'Some random trial that needs to run while images are loading -- e.g. instructions. Press any key to continue'
}
timeline.push(trial_during_which_media_needs_to_be_loaded_1)
var trial_during_which_media_needs_to_be_loaded_2 = {
	type: 'html-keyboard-response',
	stimulus: 'A second random trial that needs to run while images are loading -- e.g. instructions. Press any key to continue'
}
timeline.push(trial_during_which_media_needs_to_be_loaded_2)

var wait_until_media_is_fully_loaded = {
	type: 'call-function',
	async: false,
	func: function() {
		// critically, here I need to stop the experiment and only resume it after all promises have been successfully resolved
		jsPsych.pauseExperiment()
		PROMISES.all().then(responses => {
				jsPsych.resumeExperiment()
			})
	}
}
timeline.push(wait_until_media_is_fully_loaded)

var trial_to_run_after_media_is_loaded = {
	type: 'html-keyboard-response',
	stimulus: 'This trial must not show until all images are loaded. Press any key to continue'
}
timeline.push(trial_to_run_after_media_is_loaded)

This does not work. The trial_to_run_after_media_is_loaded runs (ergo the wait_until_media_is_fully_loaded trial finishes running) before all images are loaded. No clue why. If I add console.log statement within the PROMISES.all().then() part, the console logs only after images are loaded, yet the timeline continues!!!

An alternative I tried was to not preset the timeline, but rather set it only after promises are resolved. So the last part would be:

// an alternative to waiting and timeline setup above
var wait_until_media_is_fully_loaded = {
	type: 'call-function',
	async: false,
	func: function() {
		// critically, here I need to stop the experiment and only resume it after all promises have been successfully resolved
		jsPsych.pauseExperiment()
		PROMISES.all().then(responses => {
				jsPsych.addNodeToEndOfTimeline(trial_to_run_after_media_is_loaded)
				jsPsych.resumeExperiment()
			})
	}
}
timeline.push(wait_until_media_is_fully_loaded)

// here, this trial is created but not pushed to the timeline until -- ALLEGEDLY -- the media is preloaded
var trial_to_run_after_media_is_loaded = {
	type: 'html-keyboard-response',
	stimulus: 'This trial must not show until all images are loaded. Press any key to continue'
}

Same result as before!! I've tried using html-keyboard-reponse trial type instead of call-function -- same result! I am perplexed beyond belief! Adding some breakpoints here and there shows me that in the call-function case, then function is executed before promises are resolved (fine!) but no idea how jsPsych knows that it should continue the experiment before promises are resolved......

[nikbpetrov]

I've managed to find a working solution for this. The key for me was to 1) promise-ify the preload_image function in the jsPsych core module (e.g. see here) and 2) change it to return an array of the promises for each image.

So the module.preloadImages function now looks like this:

module.preloadImages = function(images, callback_complete, callback_load, callback_error) {

    // flatten the images array
    images = jsPsych.utils.flatten(images);
    images = jsPsych.utils.unique(images);

    var n_loaded = 0;
    var finishfn = (typeof callback_complete === 'undefined') ? function() {} : callback_complete;
    var loadfn = (typeof callback_load === 'undefined') ? function() {} : callback_load;
    var errorfn = (typeof callback_error === 'undefined') ? function() {} : callback_error;

    if(images.length === 0){
      finishfn();
      return;
    }
    function preload_image(source){
      return new Promise((resolve, reject) => {
        var img = new Image();

        img.onload = function() {
          n_loaded++;
          loadfn(img.src);
          resolve(img.src)
          if (n_loaded === images.length) {
            finishfn();
          }
        };

        img.onerror = function(e) {
          errorfn({source: img.src, error: e});
          reject(img.src)
        }

        img.src = source;

        img_cache[source] = img;
        preload_requests.push(img);
      })
    }

    let img_promises = []
    for (var i = 0; i < images.length; i++) {
      let current_img_promise = preload_image(images[i]);
      img_promises.push(current_img_promise)
    }
    return img_promises
  };

This then allows me to save the promises from the preloadImages function and check if they are resolved later, e.g.

let images = ['img_1.jpg', 'img_2.jpg']
let current_experiment_img_promises = jsPsych.pluginAPI.preloadImages(images)

var wait_until_media_is_fully_loaded = {
	type: 'html-keyboard-response',
	stimulus: '<div class="fixation-cross">Wait until the images are loaded.</div>',
	choices: jsPsych.NO_KEYS,
	on_load: function() {
		Promise.all(current_experiment_img_promises).then(() => {
			jsPsych.finishTrial()
		})
	}
}

@becky-gilbert @jodeleeuw This promise-ifying seems to me like a decent solution and might be a useful addition to jsPsych, provided, of course, that there isn't a better workaround that I am not aware of. Of course, one can do with better error handling than mine, but it's a decent start.

[becky-gilbert]

Hi @nikbpetrov, sorry for the very slow response, and thanks for sharing your solution to this! I'm guessing other researchers would be interested in the ability to preload in the background, so it's probably worth promise-ifying the preloading in some way to make that possible. I'm not an expert in promises, but the way you've done this looks straightforward to me. Perhaps we could also wrap the check for whether all promises have been resolved in a jsPsych pluginAPI function, for convenience. Feel free to submit a pull request, otherwise I'll leave this issue open so that we remember to implement it (with proper credit to you 😃). Thanks again!

[nikbpetrov]

Hi @becky-gilbert, I was just revisiting this in a different project and have a few more thoughts that I hope you find helpful.

It might be worth making this asynchronous loading as easy to implement as possible. Imagine that you have 1000 trials, split into 5 blocks and each trial needs to load 5 images - that's a lot of preloading. One way to resolve this is to create a Queue that would preload the images continuously in the background but have a mechanism to stop the experiment at any point and wait for preloading of a specified number of images.

For instance, you register the preloading of the 5000 images at the beginning of the experiment and just before your experiment starts you have an instructions trial, on which you set its wait_for_media_preload parameter to 1000, which will halt the experiment until the first 1000 images are loaded. Note that the optional global parameter wait_for_media_preload is just one (perhaps suboptimal) mechanism -- up to you to decide when/how you halt (maybe a pluginAPI function that is run in any trial's on_start event?).

From the backend perspective, all this parameter needs to do is add another trial that looks something like this:

let wait_for_images_to_preload = {
	type: jsPsychHtmlKeyboardResponse,
	stimulus: 'Please wait until the experiment loads...',
	choices: "NO_KEYS",
	on_load: function() {
		let trial_start = new Date();
		jsPsych.pluginAPI.setTimeout(function(){
		    document.getElementById('jspsych-html-keyboard-response-stimulus').innerHTML = '<p>The experiment is taking too long to load...</p><p> Please refresh the page by pressing Ctrl + F5 (Windows) or Cmd + Shift + F5 (Mac)</p>'
		}, 60000);
		Promise.all(IMAGES_PROMISES).then(() => {
			let trial_end = new Date();
			let seconds_since_trial_start = (trial_end.getTime() - trial_start.getTime()) / 1000;
			if (seconds_since_trial_start < 2) {
				jsPsych.finishTrial()
			} else {
				document.getElementById('jspsych-html-keyboard-response-stimulus').innerHTML = 'Starting in 2 seconds...'
				jsPsych.pluginAPI.setTimeout(function() {
					jsPsych.finishTrial()
				}, 2000)
			}				
		})
	},
	simulation_options: {
		simulate: false
	},
}

You can see, I've added some additional stuff to the above example of wait_until_media_is_fully_loaded but the barebones of this trial are in my previous comment.

Another thing to consider is whether the asynchronous queue affects the experiment runtime. In some experiments, timing is very important (e.g. masking paradigms, where the difference between 1 and 2 frames is critical) so not sure if network load affects browser local speed.

[bjoluc]

Hi @nikbpetrov, reading over this again, I realize asynchronous loading sounds like a perfect fit for an extension. It might be initialized with a list of paths and, when added to a trial, pause the experiment at (/after?) the trial until a given set of paths has been loaded. Do you think that would be a reasonable abstraction and would it work in your use case?

Re promisifying asynchronous pluginAPI functions: I've dreamed of this for a long time, and I'd love to bake it into v8, but we haven't yet discussed it. @jspsych/core WDYT?

[nikbpetrov]

Hi @bjoluc , you got it right.

One relatively minor thing to consider is how to "pause the experiment": it can't just be a blank screen with jsPsych.pauseExperiment and some loading information would be nice -- e.g. like the example in the docs here. Though if you add an additional trial from the backend, this needs to be handled carefully - it should probably not be saved in the data (imagine someone counts trials before trial N but fails to realise that there is an additional one added). Maybe something akin to the preload plugin that incorporates promises.

Though, if you actually end up promisifying core jsPsych functionality, the above might be solved much more easily (e.g. initialise the preload plugin with command: "init" and then pause with the same plugin but this time set its command parameter to command: "wait", in a very similar way to how the jspsych pavlovia plugin works (original plugin here, improved version by Thomas Pronk here, and my version with a new "save" command in addition to "init" and "finish" here*).

*I guess one interesting thing to add for your consideration in promisify-ing is the ability to attempt multiple times to execute a request. This is mentioned in the "next steps" section of the last link of my version and I have later developed that for my own projects. The pattern is generically described here (version 1, with delay), but also happy to share my jspsych plugin file if needed.