Remotion + Rive + Referenced Assets

[zoephilipps]

Feature Request 🛍️

@JonnyBurger Thanks for the Rive support in #4069! I'm curious what the best approach would be for using Rive + Remotion with referenced or dynamic assets?

I see that ref support was added with the above PR, but I'm unclear on the use case of that or if that could be leveraged here.

I can make it work with useRive, but since I'm not using RemotionRiveCanvas, that doesn't sync Rive to Remotion's clock. Is there an easy solution for that?

Use Case

For our use case, we're trying to make images dynamic, and we'll eventually need to do the same thing with a custom typeface.

Update: We also need useRive support for dynamic text.

Possible Solution

Not sure.

[JonnyBurger]

You can already change the text run using ref.current.getArtboard(): https://rive.app/community/doc/text/docn2E6y1lXo#low-level-api-usage

I'm adding support for custom asset loading in the next version! Refer to the updated docs.

[zoephilipps]

Hey @JonnyBurger, thanks for this. I was able to get it almost working with the changes in #4091. However, now I'm running into this issue, and I have no idea why it's happening:

The issue also happens in @rive-app/[email protected].

Since it seems to be complaining about instances of the same class, maybe the issue is that there are multiple instantiations of something when there should only be one?

Here's the relevant code:

import {useCallback, useRef} from "react";
import {staticFile} from "remotion";
import {RemotionRiveCanvas, RiveCanvasRef} from "@remotion/rive";
import {FileAsset, ImageAsset} from "@rive-app/canvas-advanced";
import {decodeImage} from "@rive-app/react-canvas";

export const Scene02 = () => {
    const ref = useRef<RiveCanvasRef>(null);

    const assetLoader = useCallback((asset: FileAsset, bytes: Uint8Array) => {
        if (!ref.current) return false;

        console.log("Asset properties to query", {
            name: asset.name,
            fileExtension: asset.fileExtension,
            cdnUuid: asset.cdnUuid,
            isFont: asset.isFont,
            isImage: asset.isImage,
            isAudio: asset.isAudio,
            bytes,
        });

        // If the asset has a `cdnUuid`, return false to let the runtime handle
        // loading it in from a CDN. Or if there are bytes found for the asset
        // (aka, it was embedded), return false as there's no work needed here
        if (asset.cdnUuid.length > 0 || bytes.length > 0) {
            return false;
        }

        if (asset.isImage && asset.name === "80895011_1718952048763209_r") {
            fetch("https://picsum.photos/300/500").then(async (res) => {
                console.log("doing this");
                const image = await decodeImage(new Uint8Array(await res.arrayBuffer()));

                /**
                 * vvvvvvvvvvv ERROR BELOW vvvvvvvvvvvv
                 */
                (asset as ImageAsset).setRenderImage(image);

                // You could maintain a reference and update the image dynamically at any time.
                // But be sure to call unref to release any references when no longer needed.
                // This allows the engine to clean it up when it is not used by any more animations.
                image.unref();
            });

            return true;
        }

        return false;
    }, []);

    return (
        <RemotionRiveCanvas
            artboard="02"
            src={staticFile(`nextassets/video/untitled.riv`)}
            assetLoader={assetLoader}
            enableRiveAssetCdn={false}
            ref={ref}
        />
    );
};

Rive file: untitled.riv.zip

The rest of the code is the same from #4083 except all the Remotion version numbers have been bumped to 4.0.181.

[JonnyBurger]

@zackphilipps Does your PR fix this issue?

[zoephilipps]

@JonnyBurger No, it simply allows the assetLoader function to be called and executed.

[JonnyBurger]

@zackphilipps You say the issue also happens in @rive-app/canvas-advanced.

Do you think this is a Remotion issue or one in Rive? Maybe it should be reported to the Rive team instead

[zoephilipps]

You say the issue also happens in @rive-app/canvas-advanced.

@JonnyBurger I was simply saying it also happens with the next version of that package.

Do you think this is a Remotion issue or one in Rive? Maybe it should be reported to the Rive team instead

I thought this was somehow a Remotion issue since the useRive hook worked without a problem. But, since useRive comes from @rive-app/react-canvas, the issue may be limited to @rive-app/canvas-advanced. I can look into it further on Monday.

[zoephilipps]

@JonnyBurger Well hang on, why does their example work then?

Update: Ah, it uses @rive-app/canvas, not @rive-app/canvas-advanced. I will file an issue with them.

[JonnyBurger]

Closing because it was transferred to external and seems to have been resolved