Sound support

[vlidholt]

From what I can tell, there is currently no support for playing sounds? Any advice as where would be a good starting point for making an implementation?

[ardera]

Flutter doesn't have official audio support as of now. (ref)

The most popular plugin for audio support seems to be audioplayers. That plugin seems to have no internal checks whether it's running on an officially supported platform. It just opens the method channel xyz.luan/audioplayers and calls methods on it. So if you wanted to add linux support to it, you don't need to modify any of the code in the package. You can just implement all the method calls in C code inside flutter-pi, and it'll work.

For example, look at lib/audioplayers.dart line 401. There _invokeMethod('play', {...}) is called. If you look at the implementation of _invokeMethod, it'll wrap the map arg in another map and add a playerId and mode.

Some code to listen and decode this in a flutter-pi flutter-pi plugin would look like the following:

// sadly, because of some stupid design decisions I made when starting this project, we sometimes need to rely
// on global state. 
struct {

} my_audio_plugin;

int on_play(struct std_value *arg, FlutterPlatformMessageResponseHandle *responsehandle) {
    struct std_value *temp;
    int64_t player_id;

    if (!STDVALUE_IS_MAP(*arg)) {
        return platch_respond_illegal_arg_std(
            responsehandle,
            "Expected `arg` to be a map."
        );
    }
    
    temp = stdmap_get_str(arg, "playerId");
    if (((temp != NULL) && STDVALUE_IS_INT(*temp)) {
        player_id = STDVALUE_AS_INT(*arg);
    } else {
        return platch_respond_illegal_arg_std(
            responsehandle,
            "Expected 'arg['playerId']` to be an integer."
        );
    }

    // ...

    return platch_respond_success_std(responsehandle, &STDINT64(1));
}

int on_method_call(char *channel, struct platch_obj *object, FlutterPlatformMessageResponseHandle *responsehandle) {
    // we received a method call on "xyz.luan/audioplayers"
    // check the name of the method.

    // a struct platch_obj is basically a decoded platform message.
    // See `platformchannel.h` for more documentation on that.
    
    char *method_name = object->method;
    
    if (strcmp(method_name, "play") == 0) {
        return on_play(object->, responsehandle);
    } else {
        // respond with not implemented.
        // this will trigger a `MissingPluginException` in the dart VM.
        return platch_respond_not_implemented(responsehandle);
    }
}

int my_audio_plugin_init(void) {
    // register a callback to be invoked on the programs main thread when
    // a platform message is received on channel "xyz.luan/audioplayers"
    // and automatically decode it as a standard method codec method call.
    // (== method channel)
    plugin_registry_set_receiver("xyz.luan/audioplayers", kStandardMethodCall, on_method_call);
    return 0;
}

int my_audio_plugin_deinit(void) {
    plugin_registry_remove_receiver("xyz.luan/audioplayers");
    return 0;
}

You'll find some documentation in the header & source files. If you have any questions, feel free to ask.

The platform channel and plugin registry APIs are (99% at least) thread-safe. You also don't need to reply to the method call synchronously. You can take your time, maybe pass the responsehandle to some worker thread, and let that worker thread respond to the platform message using platch_respond_XXX.

As to the implementation of the actual audio playback: I'd suggest using gstreamer. I'm actually using gstreamer for video decoding for another plugin (which is in a branch I haven't pushed yet), so this way we can reuse some things and don't add more dependencies.

Ngl, gstreamer is kinda complicated and not that easy to work with, but it supports hardware decoding and, in contrast to VLC player for example, supports zero-copy and dmabufs. Dmabufs are good because they have integrated support in EGL. You can just import a dmabuf to an OpenGL texture without any memory copied. Generally, zero-copy is good to have on embedded devices like the Pi, since memory bandwidth is kinda scarce.

[jtkeyva]

@ardera awesome stuff!

I'd suggest using gstreamer. I'm actually using gstreamer for video decoding for another plugin (which is in a branch I haven't pushed yet), so this way we can reuse some things and don't add more dependencies.

does video and audio playback stable as of now? you still using gstreamer? have you pushed the branch?

thanks

[BigGitWorld]

@ardera How do I have to use your code to be able to use audioplayers in a flutter-pi project?

[ardera]

audio player plugin merged a few weeks ago, you should be able to use the audioplayers package now.

[ravi-dhorajiya]

gstreamer

Hello @ardera , I've added audioplayers package with the latest version and it is still not working on pi os. any idea?

[dqbd]

When attempting to call play, the following error occurs:

[ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: PlatformException(illegalargument, Expected `arg['player_id'] to be a string., null, null)
#0      StandardMethodCodec.decodeEnvelope (package:flutter/src/services/message_codecs.dart:652:7)
#1      MethodChannel._invokeMethod (package:flutter/src/services/platform_channel.dart:310:18)
<asynchronous suspension>
#2      AudioPlayer._create (package:audioplayers/src/audioplayer.dart:144:7)
<asynchronous suspension>

[DoumanAsh]

All this time there was incorrect code and I didn't notice as I used my own fork of flutter-pi

https://github.com/ardera/flutter-pi/pull/342