Broadway
Broadway copied to clipboard
mbebenita
Acquire Audio Data
I'm using Aurora.js (and previously JSMad) to decode audio to display as a visual waveform. I'd really like to do the same thing with audio tracks out of a video file, and I figure I can do that by using mp4.js to parse a video file and then just extract the audio data, but I can't quite figure out how to do that. So, how would one go about accessing any audio data in an mp4 file?
Sorry for the late reply. The mp4.js file should be able to give you the raw audio samples, take a look at https://github.com/mbebenita/Broadway/blob/master/Player/mp4.js#L459
I would recommend looking at the spec to understand how it all works, I never dealt with the audio part myself. The Apple QuickTime File Format spec is a good place to start, it's practically the mp4 spec.
@gliese1337 hi i have no tips for you but maybe you can help me with something i am trying to find a low latency audio solution for the browser. do you have any tips for me? i also tried aurora but so far the latency is still to high. be aware i am a audio codec noob
@soliton4 "audio solution" is pretty vague. Aurora.js has always worked fine for me.
i am creating a remote desktop based on broadway. for audio i would need a way to have low latency audio transfered from the server to the client.
i tried using the output from opusenc and pass it to aurora. at first it is low in latency, but latency gets bigger and bigger. you got any ideas for me @gliese1337 ?
@soliton4 Since you control the server-side encoding, I would probably just dump aurora entirely and transcode to some format that the browser can handle natively. You can then read in fixed-size chunks of audio to package into Blobs which get fed into native HTML5 Audio elements via Blob URLs. That'll at least give you fixed latency based on whatever frame size you use for making your audio blobs.
If that's still unacceptable, I'd just try using some other codec with Aurora (again, because you control the server-side transcoding). It might just be that the Opus decoder is slower than the others, or has unresolved memory leak bugs or something; it is a lot newer than, say, mp3.js, after all.
On another note more directly related to this issue, I've recently become much more familiar with mp4 file structure due to using exactly that technique of simulating streaming with fixed latency by packaging chunks of a stream into file Blobs in order to play HLS streams on non-apple devices. That involves sticking audio into an MP4 file rather than pulling it out, but the pulling out part should be relatively straightforward. No promises, but I might get around to implementing an Aurora-based audio extension for Broadway in the near-ish future, since it's pretty similar to what I'm getting paid for these days.
Hey @gliese1337, any news regarding the last note? did u manage to pull the audio data?
thanks.
@bsliran Enhancing my waveform display to work with video files has never been high enough priority for me to get paid for yet, and I haven't had the free time in the last year to tackle it as a side-project. But it looks like it should be pretty straightforward, and I might actually have a legitimate product-driven reason to do it in the next couple of months.
I've been struggling to find a way to get Broadway to play audio straight from the mp4 file. My main issue is that the native HTML5 video element, no matter what's done, always has a slight pause when looping videos and Broadway might be the solution I've been looking for for years. Would be great to see some better usage docs as to how to do this.
