🔧 Creating custom frame processor plugin for Expo Modules

[mgulfam0722]

How were you trying to build the app?

Hi, I am completely new to native programming and having a really difficult time creating a custom frame processor for iOS and Android in Expo Modules, since Expo EAS doesn't support modifications to ios/ and android/ native directories it has to be developed using Expo Module but the react-native-vision-camera did not document steps for Expo Modules.

Any help ,guidance is highly appreciated. Thanks and regards

Full build logs

Didn't build yet, as there are no steps to build custom plugin in Expo Modules.

Project dependencies

"dependencies": {
    "@expo/vector-icons": "^14.0.2",
    "@hookform/resolvers": "^5.0.1",
    "@infinitered/react-native-mlkit-core": "^3.1.0",
    "@infinitered/react-native-mlkit-face-detection": "^3.1.0",
    "@react-native-community/blur": "^4.4.1",
    "@react-native-community/datetimepicker": "8.2.0",
    "@react-native-masked-view/masked-view": "0.3.2",
    "@react-navigation/bottom-tabs": "^7.2.0",
    "@react-navigation/native": "^7.0.14",
    "apisauce": "^3.1.0",
    "expo": "~52.0.42",
    "expo-checkbox": "~4.0.1",
    "expo-dev-client": "~5.0.19",
    "expo-font": "~13.0.4",
    "expo-linear-gradient": "~14.0.2",
    "expo-router": "~4.0.20",
    "expo-secure-store": "~14.0.1",
    "expo-splash-screen": "~0.29.22",
    "expo-status-bar": "~2.0.1",
    "expo-system-ui": "~4.0.9",
    "react": "18.3.1",
    "react-dom": "18.3.1",
    "react-hook-form": "^7.56.0",
    "react-native": "0.76.9",
    "react-native-gesture-handler": "~2.20.2",
    "react-native-otp-entry": "^1.8.4",
    "react-native-reanimated": "~3.16.1",
    "react-native-safe-area-context": "4.12.0",
    "react-native-screens": "~4.4.0",
    "react-native-select-dropdown": "^4.0.1",
    "react-native-svg": "15.8.0",
    "react-native-toast-message": "^2.3.0",
    "react-native-vision-camera": "^4.6.4",
    "react-native-web": "~0.19.13",
    "react-native-webview": "13.12.5",
    "react-native-worklets-core": "^1.5.0",
    "zod": "^3.24.3"
  }

VisionCamera Version

4.6.4

Target platforms

iOS

Operating system

MacOS

Can you build the VisionCamera Example app?

Yes, I can successfully build the Example app here

Additional information

• [x] I am using Expo
• [x] I have enabled Frame Processors (react-native-worklets-core)
• [x] I have read the Troubleshooting Guide
• [x] I agree to follow this project's Code of Conduct
• [x] I searched for similar issues in this repository and found none.

[maintenance-hans[bot]]

Guten Tag, Hans here! 🍻

Thanks for your detailed issue report. I understand creating a custom frame processor can be challenging, especially with Expo Modules. While I see your concern about documentation, I recommend checking out the Expo documentation for guidance on native modules.

However, since the issue lacks specific logs and reproducible steps, I cannot provide direct assistance right now. If you face any build errors, please make sure to provide logs from Xcode or adb logcat, so mrousavy can troubleshoot effectively.

If you're keen on getting more attention to this issue, consider sponsoring the project with this lovely link: Sponsor mrousavy. It helps improve responsiveness!

Feel free to reach out with more information!

Note: If you think I made a mistake, please ping @mrousavy to take a look.

[christophemenager]

Hi @mgulfam0722, I am currently looking to do the exact same thing.

1. I tried using the same Frame processor plugin template for ios but it's not of type "Module" so I removed the module from expo-module.config.json This does not work as the module ends up not being registered

2. I tried to create an expo module from scratch that would expose a method to analyze a frame. This could work with some work, but the type Frame as input is not compatible with expo module and you would have to write a custom type converter which seems quite complex.

At the moment I am stuck, I can't figure out a way to use CNG on expo with custom frame processor plugins

[musabgulfam]

Hi @christophemenager, exactly same problem here, the docs do not provide any resource for integration with Expo Modules. I tried with standalone Expo Module, but frame processor won't get registered. The only option available is to "eject" from expo and utilise the frame processor in react native cli, which is not best suited when Expo Router, Expo EAS and a whole lot of Expo modules are integrated.

[christophemenager]

I found a way to make it work, but I have sometimes an error at runtime (I have to kill the app and it works again)

Solution found to implement a frame processor plugin with expo modules API

Context

ObjectDetectionFrameProcessorPluginis the frame processor plugin that extends the FrameProcessorPlugin as described in the docs

🍏 For IOS

This was the most tricky part as the documentation does not describe how to manually register a plugin

import ExpoModulesCore
import VisionCamera

public class ObjectDetectionFrameProcessorModule: Module {
  public func definition() -> ModuleDefinition {
    Name("ObjectDetectionFrameProcessor")

    OnCreate {
      FrameProcessorPluginRegistry.addFrameProcessorPlugin(
        "searchObjectsInFrame", 
        withInitializer: { proxy, options in
          return ObjectDetectionFrameProcessorPlugin(proxy: proxy, options: options)
        }
      )
    }
  }
}

** ⚠️ Warning: I keep getting this error at runtime when I try to reload the app**

NativeUnimoduleProxy has no setter or ivar for its
bridge, which is not permitted. You must either
@synthesize the bridge property, or provide your
own setter method.

@mrousavy Do you think my iOS implementation is ok (I do not use the default swift export and instead I manually register the plugin? Did I miss something? You answer could help many others and I can make a PR to update the docs if you think this is the right way

🤖 For Android

This part is easy as the documentation is clear on how to manually register a frame processor plugin

package expo.modules.objectdetectionframeprocessor

import expo.modules.kotlin.modules.Module
import expo.modules.kotlin.modules.ModuleDefinition

import com.mrousavy.camera.frameprocessors.FrameProcessorPluginRegistry
import expo.modules.objectdetectionframeprocessor.ObjectDetectionFrameProcessorPlugin

class ObjectDetectionFrameProcessorModule : Module() {
  override fun definition() = ModuleDefinition {
    Name("ObjectDetectionFrameProcessor")

    OnCreate {
      FrameProcessorPluginRegistry.addFrameProcessorPlugin("searchObjectsInFrame") { proxy, options ->
        ObjectDetectionFrameProcessorPlugin(proxy, options) 
      }
    }
  }
}

This works fine with no errors on android

[christophemenager]

Ok, I found a workaround that works with hot reloading:

import ExpoModulesCore
import VisionCamera

public class ObjectDetectionFrameProcessorModule: Module {
  private static var isRegistered = false
  public func definition() -> ModuleDefinition {
    Name("ObjectDetectionFrameProcessor")

    Constants([
      "analyzeMethod": "searchObjectsInFrame"
    ])

    OnCreate {
      if !ObjectDetectionFrameProcessorModule.isRegistered {
        // Prevents issue with hot reloading: NativeUnimoduleProxy has no setter or ivar for its bridge
        FrameProcessorPluginRegistry.addFrameProcessorPlugin(
          "searchObjectsInFrame", // The name JS will use
          withInitializer: { proxy, options in
            return ObjectDetectionFrameProcessorPlugin(proxy: proxy, options: options)
          }
        )
        ObjectDetectionFrameProcessorModule.isRegistered = true
      }
    }
  }
}

Same principle for android

Not sure this is the best way, but with this, I managed to extract my custom frame processor plugins to expo modules and reactivated CNG in my expo project 🎉

[dmausttruefit]

@christophemenager Thank you for the examples! Do you happen to have a more complete example to follow along? Anything you've learned in the last 3 weeks that could have influenced your current solution?

[christophemenager]

Do you happen to have a more complete example to follow along?

I can detail a bit more but I would prefer to document it properly @mrousavy would you accept a documentation PR that adds a section about creating a local frame processor plugin using expo modules?

Anything you've learned in the last 3 weeks that could have influenced your current solution? Nope, my solution is working fine production since 2 weeks :)

[mantu-bit]

@christophemenager i tried your approach but it didn't worked. I am stuck in a critical project since last 2 days. Have you got any other solution then this ?