ensure CreateSynth().prime() resolves by using timeouts to avoid gett…

[syntithenai]

I'm working on a tunebook using React and abcjs => https://github.com/syntithenai/abc2book

To minimise users waiting, I am priming the audio when the abc prop to the react component changes without respect to mouse click.

The AudioContext is fine to be initialised and loaded without clicking, it's just resuming that quietly fails.

When resuming without having clicked on the page, resume hangs and the promise returned by the prime method never resolves.

I have added setTimeouts of 500ms to resolve in case resuming fails.

[paulrosen]

Sorry for the delay in responding. I need to test this a bit. My first thought is that you could get the audiocontext in advance on any click that the user does. That would give you time to preload the sounds without this change. But iOS can be really funny about this - sometimes even when you get an audiocontext on an iPhone correctly if you don't use it for a minute it gets suspended. And if a call comes in then it gets set to "interrupted". So your change might be necessary.

But I was also thinking that the slow part is probably the network. I could add a pre-load step were the mp3s are loaded in advance and only processed at the last minute.

[syntithenai]

Hey Paul, thanks for getting back so quickly. Sometimes I take weeks to respond ;)

Waiting for the first click isn't necessary and detracts from the user experience. Audio rendering on big scores can take 20s+ so getting stuck into the job as early as possible without waiting for the user is good.

In my app, the play button goes green (audio generated) within some seconds without user interaction.

It's not so much the getting the audio context (which as per your note about iOS has hazards), it's the calling of resume on the context in getSynth that binds the object creation to resuming the audio. Perhaps better unbound into different functions but I'm happy with an exit strategy (timeouts)

I have looked at increasing the speed of the audio rendering. Because I am prerendering and I'm offering editing, the prerenders can start to stack up every time I make a change and the whole UI gets sluggy.

I've come up with a few strategies to help.

• hashing of changes so React only renders when important fields in the tune change (voices/tempo/...)
• setTimeout on render to prevent the render function being called more than once per second.
• local copy of the soundfont files for piano. I'll include these in the app manifest so that rendering works offline too.
• caching of rendered audioBuffers so I can skip the second stage of placing the mp3s into the audioContext.
• I looked at web workers which could be applied to the init phase but won't help with prime phase because AudioContext isn't available in a WebWorker.

The init stage is under a second now, restoring rendered audio from cache takes under a second so once a tune has been loaded once, it's a couple of seconds.

The first render/prime can still be pretty lengthy. I've wondered if the priming stage can be rendered into a plain serialisable ArrayBuffer so it can be placed into a Web Worker. More research required.

food for thought

cheers

Steve

[paulrosen]

Sorry for the loooooong delay. I think I'm going to tackle this by moving the resume somewhere else. That seems more reliable than the time out.

Thanks a lot for opening the issue and helping me see the problem.