precise playback of small samples in sequence has unexpected noise

[coreyoconnor]

(WIP ticket - multiple unknowns still)

The wes emulator tries to queue audio samples one right after another:

• https://gitlab.com/coreyoconnor/wes/-/blob/master/gui/shared/src/main/scala/wes/gui/Main.scala?ref_type=heads#L172

Which it does so by clips as sample buffers to queue for output.

The expectation is that if sample buffers

A, B, C

and then

D, E, F

are queued the resulting samples output (assuming they are queued soon enough) would be:

A, B, C, D, E, F

in actual there appear to be discontinuities in the sample playback.

One contributing factor might be the loss of precision in sample counts from Double/Int conversions. Specifically I wonder:

• https://github.com/JD557/minart/blob/master/core/shared/src/main/scala/eu/joaocosta/minart/audio/AudioClip.scala#L132

    val duration = data.size / sampleRate
    AudioWave.fromIndexedSeq(data, sampleRate).take(duration)

followed by

• https://github.com/JD557/minart/blob/master/core/shared/src/main/scala/eu/joaocosta/minart/audio/AudioWave.scala#L68

def getAmplitude(t: Double): Double =
      data.applyOrElse((t * sampleRate).toInt, (_: Int) => 0.0)

Would result in a sample that is not actually the final sample when t == duration.

TODO:

• [x] dump the nes samples direct to file and see if there are discontinuities there

[coreyoconnor]

I've dumped the raw samples to a wav file: The playback discontinuities are not in the raw samples. When the audio is working (still some emulation bugs) the audio is clean.

[JD557]

Thanks for this Indeed, I noticed some weird issues with the audio queues, but I never got any easy way to reproduce it.

I think there's also one issue with the Oscillators in that, IIRC, if you join two clips generated by oscillators the phase won't match won't match, and I'm not entirely sure how to fix that :/ (Although I think you are already handling that with the drop and take logic)

[coreyoconnor]

Audio is definitely better but... Not quite right. I have a "record the raw audio" option. Which, after the JVM build hits steady state (60 fps and above 100% emulated performance) sounds really close but I still hear pops that do not show up in the raw audio.

wes-gui-jvm run --output-volume 0.3 --save-audio metroid.raw Metroid.nes

However, is that a minart thing? Is that a problem with how wes schedules the audio output? If wes misses scheduling the next audio chunk by a wee bit: pop. The emulation is also not-quite-right. Which makes just listening to this not very nice.

I'm pretty sure I can capture the monitor of the audio output from minart and compare that against the raw audio saved by the emulator direcly. That is probably easier to analysis than my hearing haha

[coreyoconnor]

I don't think the noise I'm hearing is to do with float/int conversion. Specifically I checked that the last index sampled for a AudioWave would be greater than the last - which results in a 0.

I tried adding this, presumably exhaustive, set of assertions to check if this ever occurs:

      val sampleCount = samples.size
      val sampleRate = state.apu.sampleBuffer.sampleRate.toDouble

      val audioClip = AudioClip.fromIndexedSeq(samples, state.apu.sampleBuffer.sampleRate.toDouble)
      val out = audioClip.map(_ * config.outputVolume)

      val endSample = (audioClip.duration * sampleRate).toInt
      assert(endSample == sampleCount, s"$endSample != $sampleCount")

      val stepSize = 1.0 / sampleRate

      var position = 0
      var c = 0
      while(c < endSample) {
        val p = c * stepSize
        val i = (p * sampleRate)
        assert(i < sampleCount, s"${i} >= ${sampleCount}")
        c += 1
      }

      audioPlayer.play(out)

As best I can tell that captures all the int/double conversions used to play back a sample.

The assertions did not fail. So I don't think any int/double conversion oddness is at play here.

[JD557]

I haven't had a lot of time to work on this, but I think there might be an issue with how the current audio thread is implemented.

I have a local branch where I make it so that the audio thread logic is configurable (i.e. spawn the thread once and never kill it), but I haven't had the time to properly test it.