graphics: play a movie or a video

[hajimehoshi]

[hajimehoshi]

I'm not sure how feasible this would be 🤔

[KeitoTobi1]

MP4 Library. https://github.com/abema/go-mp4

[hajimehoshi]

Thanks, but I think this doesn't include codecs.

[Zyko0]

Is going ffmpeg route an option ? I've seen quite a few go bindings, but I'm afraid this would add some CGO again, idk if it's an issue

[hajimehoshi]

Cgo is not an option especially for Windows, unfortunately. Another issue is that ffmpeg is LGPL and even if we had pure Go version of ffmpeg, this would conflict with Ebiten's license.

[hajimehoshi]

https://github.com/CrazyInfin8/mpg-go

[mrg0lden]

One side note regarding CGo issues, I think some can be mitigated using WASM based bindings and pure Go WASM runtime like this one: https://github.com/tetratelabs/wazero

For most C libraries which are portable, I think this approach works.

[SolarLune]

https://github.com/CrazyInfin8/mpg-go

This seems like a fine approach. Playing back a variety of formats would be ideal, but is understandably difficult to manage - it might be fine to simply create a script or tool to invoke FFMPEG on the developer's computer and convert videos to the correct format. Either this could be distributed as part of ebitengine's toolset, or simply mentioned on a wiki / knowledgebase for the developer to run as a script (like the Linux game building script).

One side note regarding CGo issues, I think some can be mitigated using WASM based bindings and pure Go WASM runtime like this one: https://github.com/tetratelabs/wazero

For most C libraries which are portable, I think this approach works.

I'm not knowledgeable on WebAssembly, so forgive me if I'm not understanding how this works, but if I understand you correctly, your idea is to compile FFMPEG into a web assembly binary (?), use wazero to run the binary, use bindings to call FFMPEG in the web assembly runtime to decode videos, and finally get that data and play the audio and display the frames?

[mrg0lden]

@SolarLune Yes, I'm not sure how different the performance will be. (The WASM runtime compiles the code to machine code, also the execution part is written in assembly, so I assume it is performant, or at least as performant as CGo) I tried it on one C library, and the least I got is portability and stability. So I think at least these two benefits do exist.

[hajimehoshi]

https://github.com/gen2brain/mpeg

[gen2brain]

Hi, I added an Ebitengine example here https://github.com/gen2brain/mpeg/blob/main/examples/player-eb/main.go. There are some issues though, the audio is crackling. I tried to play the SDL example with the S16 format I added and it works fine there. Also, if I try to seek, the memory will go crazy, 1G and more. I added some fake Seek so I can also seek an audio player, but that didn't change anything.

[hajimehoshi]

The latest Ebitengine v2.5 should mitigate the issue, but I failed go mod tidy

$ git diff
diff --git a/examples/player-eb/go.mod b/examples/player-eb/go.mod
index 61c0163..b1f869e 100644
--- a/examples/player-eb/go.mod
+++ b/examples/player-eb/go.mod
@@ -6,7 +6,7 @@ go 1.19
 
 require (
        github.com/gen2brain/mpeg v0.1.0
-       github.com/hajimehoshi/ebiten/v2 v2.4.8
+       github.com/hajimehoshi/ebiten/v2 62cbe99a274dd783ace736b842cbd2353e16af19 
        github.com/jfbus/httprs v0.0.0-20190827093123-b0af8319bb15
 )

$ go mod tidy
go: downloading github.com/hajimehoshi/ebiten/v2 v2.5.0-alpha.4.0.20221025043734-62cbe99a274d
player-eb imports
        github.com/hajimehoshi/ebiten/v2 imports
        github.com/hajimehoshi/ebiten/v2/internal/ui imports
        golang.org/x/mobile/app imports
        golang.org/x/exp/shiny/driver/gldriver: ambiguous import: found package golang.org/x/exp/shiny/driver/gldriver in multiple modules:
        golang.org/x/exp v0.0.0-20190731235908-ec7cb31e5a56 (/Users/hajimehoshi/go/pkg/mod/golang.org/x/[email protected]/shiny/driver/gldriver)
        golang.org/x/exp/shiny v0.0.0-20221025133541-111beb427cde (/Users/hajimehoshi/go/pkg/mod/golang.org/x/exp/[email protected]/driver/gldriver)
player-eb imports
        github.com/hajimehoshi/ebiten/v2 imports
        github.com/hajimehoshi/ebiten/v2/internal/ui imports
        golang.org/x/mobile/app imports
        golang.org/x/exp/shiny/screen: ambiguous import: found package golang.org/x/exp/shiny/screen in multiple modules:
        golang.org/x/exp v0.0.0-20190731235908-ec7cb31e5a56 (/Users/hajimehoshi/go/pkg/mod/golang.org/x/[email protected]/shiny/screen)
        golang.org/x/exp/shiny v0.0.0-20221025133541-111beb427cde (/Users/hajimehoshi/go/pkg/mod/golang.org/x/exp/[email protected]/screen)

Related: https://github.com/hajimehoshi/ebiten/issues/2376

[tinne26]

@gen2brain I made a couple tests, and the audio issues may not be directly related to Ebitengine. I used some code to capture the audio being served by mpg, then saving it to a file at the end, and then using another Ebitengine program to play the captured audio in isolation, without video nor your mpeg package or anything... and it has the same artifacts. The capture was a simple wrapper that did only this:

n, err := mpgAudioReader.Read(buffer)
rawAudio = append(rawAudio, buffer[0 : n]...)

So I'd suspect something with the S16 format, or maybe Ebitengine trying to read too far ahead in advance at the start (it tends to issue two initial buffer reads) and that causing some issue with data not being yet ready (or not, haven't really checked how your audio streaming code works, and I also played around with SetAudioLeadTime and didn't help, so maybe not...). I also tested that it wasn't anything related to the number of channels or LSB vs MSB. Didn't find anything there either.

Regarding Seek behavior, I replaced ebiten.IsKeyPressed with inpututil.IsKeyJustPressed, and it seemed to work ok for me. So maybe you didn't notice it was retriggering many times per second due to IsKeyPressed returning true for as long as you keep pressing the key.

By the way, awesome work, this is kind of a big deal for Ebitengine, so you are making a lot of people happy!

[gen2brain]

@tinne26 Thanks, I did not know about IsKeyJustPressed, I didn't use Ebitengine before, and I suspected something is wrong there because I could not toggle fullscreen, I will change that soon. About the S16, I don't know, as I said I tested with SDL (you can choose the audio format there) and tried the example and it worked, so I ruled that out. It can also be related to the issue that was pointed out, about lock/unlock before Read, because the decoder will definitely read from its own Reader during decoding.

[gen2brain]

Yes, sorry, just to point out this, the decoder will preallocate the memory for the audio sample, and it should be used in the callback function, i.e. write the received sample to buffer. That is usually nice and enough for many libraries. In this case, I just added a bytes.Reader that reads that memory and seeks to begin if the length is 0, that is probably not the right way. There is also the other way, to call manually DecodeAudio()/DecodeVideo() instead of Decode() and sync audio/video manually, but I also didn't figure out where and how should I use those instead.

[tinne26]

So, if I understand correctly, the issue is that when the callbacks are set, data should be consumed there. The fact that we are using both the audio reader directly and Decode() means we are interacting with the audio decoder from two different places and that's what causes the artifacts, probably due to races or similar on the underlying buffers, or the direct reads jumping to the start again accidentally.

If that's correct, that's great, as it means both projects are working properly and we only need to figure out the best way to sync things in Ebitengine. Don't worry if you are more unfamiliar with Ebitengine, with all that info we can totally figure it out by ourselves if you need to focus on something else.

[gen2brain]

Well, yes, the Decode() is called with the tick seconds, usually 1/60, it will call Video/Audio callbacks any number of times, for this to work you must SetAudioLeadTime() to the duration of the buffer, that would be mpeg.SamplesPerFrame/samplerate, you should set the Ebitengine buffer size to the same duration. Also, SetVideoEnabled/SetAudioEnabled functions are all related to Decode(). In the Ebitenengine case, the audio callback function doesn't do anything, it is defined only so Decode() could decode audio (doesn't work without callback). With DecodeVideo/DecodeAudio, they just decode and return one single audio/video frame.

Also, just to mention, this is of course rewrite of the C library, everything is single-threaded, there are no go routines, etc. Both audio and video constantly check if there are more bits with the has() function that will try to load more from the reader if not available. This I guess is a perfect case for some producer/consumer pattern or something like that, but for now I didn't touch that. I did learned a lot but I am not expert here, so any help is welcome.

[hajimehoshi]

@gen2brain Do you have any suggestion about this issue? https://github.com/hajimehoshi/ebiten/issues/110#issuecomment-1291492759

[gen2brain]

@hajimehoshi No idea, mpeg package only uses the standard library. I get the same, I wanted to try and build a master without modules, but then the problem is v2.

[hajimehoshi]

I'd avoid nested go.mod if possible. You can move the examples to other repositories or simply remove child go.mod files. Both should not break backward compatibility

[gen2brain]

Ok, I removed modules, tidy now works, thanks! I also switched to inpututil.

[tinne26]

I made a small package to make it easier to manage this on Ebitengine: https://github.com/tinne26/mpegg. Still a work in progress, but it works.

The key idea is to let Ebitengine handle the audio, decoding from the mpeg as much as Ebitengine requests, and then using the audio.Player.Current() position to figure out the expected video position. When a frame is requested, we decode the video until we reach that position. I need to add some compensation for the audio, as there's always some latency due to drivers and stuff, but there don't seem to be any major problems.

I can help with the standalone example for gen2brain too, but to point to the critical code in that new package:

• The Read() method on audio_adapter.go connects the mpeg audio decoding to Ebitengine's audio player.
• The CurrentFrame() method on player.go makes the video catch up to the audio position.

[hajimehoshi]

@tinne26 Awesome! I could play the same video as @gen2brain's test video without noises!

I found an error after finishing the video:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1001612cc]

goroutine 8 [running]:
github.com/tinne26/mpegg.(*audioAdapter).Read(0xc0000a2360, {0xc00050a000, 0x44e8, 0x44e8})
        /Users/hajimehoshi/ebitengine-games/mpegg/audio_adapter.go:83 +0x32c
github.com/hajimehoshi/ebiten/v2/audio.(*timeStream).Read(0xc0004a8060, {0xc00050a000?, 0xc000526008?, 0x10014e920?})
        /Users/hajimehoshi/go/pkg/mod/github.com/hajimehoshi/ebiten/[email protected]/audio/player.go:334 +0xcf
github.com/hajimehoshi/oto/v2/internal/mux.(*playerImpl).readSourceToBuffer(0xc000506000)
        /Users/hajimehoshi/go/pkg/mod/github.com/hajimehoshi/oto/[email protected]/internal/mux/mux.go:451 +0x152
github.com/hajimehoshi/oto/v2/internal/mux.(*Mux).loop(0xc00002c200)
        /Users/hajimehoshi/go/pkg/mod/github.com/hajimehoshi/oto/[email protected]/internal/mux/mux.go:82 +0x1ce
created by github.com/hajimehoshi/oto/v2/internal/mux.New
        /Users/hajimehoshi/go/pkg/mod/github.com/hajimehoshi/oto/[email protected]/internal/mux/mux.go:46 +0x105
exit status 2

[gen2brain]

@tinne26 Feel free to modify whatever you need directly in the library. I added a lot of helper functions like RGBA(), YCbCr(), Bytes(), Pixels(), memory is preallocated for different audio formats, etc. This is one case where you definitely don't want to allocate new memory and convert something. Currently, there are no new allocations when decoding (well, anything can happen when seeking). Reader() is added just for Ebitenegine, feel free to remove it, change it, or whatever.

[tinne26]

I'll open an issue on your mpeg repository instead if that's fine with you and we can debate it there. This issue is already resolved and both mpeg and Ebitengine are doing fine, but it's true that there may be some room for improving ease of use for a couple use-cases on mpeg, and they are not related to Ebitengine itself, but rather to the "allow audio to be arbitrarily buffered and then have video catching up to a specific point in time" approach. We could make a much more compact single-file example for Ebitengine from that (or for anything else that wishes to use a similar approach).

[hajimehoshi]

At least I want to create an example to play a video (probably with https://github.com/tinne26/mpegg and/or https://github.com/gen2brain/mpeg).