tauri-apps
Add `PostSharedBufferToScript` for making streaming possible
Add PostSharedBufferToScript from webview2-rs, for making IPC streaming possible, from Rust to webview JS directly.
Useful in cases when needs to stream a lot of data efficiently, for instance video frames, audio segments... and more.
https://github.com/wravery/webview2-rs/issues/22
The thing is, shared buffers is only possible on Webview2 atm (need to confirm though) which makes it awkward to add this API atm. However, we expose the webview2 controller https://docs.rs/wry/0.34.2/x86_64-pc-windows-msvc/wry/webview/trait.WebviewExtWindows.html#tymethod.controller here so you can use it to get a reference to the Webview2Enviroment which you can use to create a shared buffer and post it to the JS side.
Another way to do stream which is supported on all platforms is through custom protocols (I guess we need to add an example for that), where the frontend makes a request to this custom protocol. We have an example in tauri repo here (This function could be copy-pasted and should work fine in wry as well). This streaming example actually serves a video from disk by setting the src of the video to the custom protocol url, then it receives the request on the Rust side where we inspect the range header and read the need bytes only.
That's good to know that I can access shared array already, even from tauri?
About the custom protocol - It looks suitable, but I'm not sure how should I use it in other use case such as sending non stop frames from webcam, to video element or directly to JS, and if it will be efficient as IPC communication
That's good to know that I can access shared array already, even from tauri?
Yes, but only on Windows and through direct Webview2 APIs.
About the custom protocol - It looks suitable, but I'm not sure how should I use it in other use case such as sending non stop frames from webcam, to video element or directly to JS, and if it will be efficient as IPC communication
It is efficient and probably the most performant data exchange mechanism we have so far. How would you use it, depends on your use-case.
- Video elements can just set its
srcproperty to the custom protocol URL and it will automatically send requests to your custom protocol callback and then you can respond appropriately. If you are capturing the webcam on Rust's side, I think you can just serve the video bytes from there as well. - Sending to JS is a bit more involved, because custom protocols are a reaction and not an action on its own, so whenever your JS needs something from the custom protocol, it has to initiate the request first. That is If you do something in Rust and at the end want to send some data to JS, then you need to store this data somewhere then eval a script that makes a request to the custom protocol where the custom protocol will serve that data.
That's good to know that I can access shared array already, even from tauri?
Yes, but only on Windows and through direct Webview2 APIs.
About the custom protocol - It looks suitable, but I'm not sure how should I use it in other use case such as sending non stop frames from webcam, to video element or directly to JS, and if it will be efficient as IPC communication
It is efficient and probably the most performant data exchange mechanism we have so far. How would you use it, depends on your use-case.
* Video elements can just set its `src` property to the custom protocol URL and it will automatically send requests to your custom protocol callback and then you can respond appropriately. If you are capturing the webcam on Rust's side, I think you can just serve the video bytes from there as well. * Sending to JS is a bit more involved, because custom protocols are a reaction and not an action on its own, so whenever your JS needs something from the custom protocol, it has to initiate the request first. That is If you do something in Rust and at the end want to send some data to JS, then you need to store this data somewhere then eval a script that makes a request to the custom protocol where the custom protocol will serve that data.
@amrbashir has kindly created an example of using a custom protocol for streaming a video file #1230 🙌
