Audio doesn't restart immediately after unpausing

[samiamwork]

Sometimes (well most of the time it seems) when playing a video via libav if a video is paused and then unpaused after a brief time, the audio won't restart right away. There will be about five seconds of silence before it returns (in sync, at lease).

Bisecting indicates that this was caused by commit 6b2e3bc04112aef01390fff09527d5ceb9621da2