Select-tf-ops not working

[llecuna]

Hi, I'm working with

• OS/OS Version: macOS 12.4
• Unity Version: 2021.3.10f1

I've transformed a tensorflow model to a tflite model using this: converter.target_spec.supported_ops = [ tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS ]

But when I execute the tflite model on Unity, this error pops up:

[asus4]

@llecuna At the moment, the select-tf-ops extension library is not included in this repository to keep the library size small. The last time I tried to build it, the size of the library increased by around 200 MB on iOS.

If many people want to use it, adding an optional package for select-tf-ops might be better.

[llecuna]

Is there any other way I can solve this for now?

[asus4]

You can build the select-tf-ops library yourself. Please refer to build_tflite.py for the build command used in this library. https://github.com/asus4/tf-lite-unity-sample/blob/master/build_tflite.py

Or, removing SELECT_TF_OPS options from your model might be more straightforward.

[llecuna]

I've seen I need NVIDIA (CUDA and cuDNN) but I'm having trouble with it since macOS no longer supports CUDA. Can I do it without it? Also, the model I'm using is SSD MobileNet V1 FPN 640x640 from this repository so I don't know how to remove those options.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[rajandeepsingh13]

@asus4 I am trying to build select-tf-ops too. I don't see a command for select-tf-ops in build_tflite.py for Android. Any hints/directions that you can provide to build the tf-lite plugin with select-tf-ops would be much appreciated.

[asus4]

hi @rajandeepsingh13, have you managed to build other Android libraries? If so, the select-tf-ops can be built with the command something like this bazel build -c opt //tensorflow/lite/java:tensorflowlite_flex.

Please refer to the original Bazel BUILD file that there is a way to export only ops used in your model.

[rajandeepsingh13]

Hi, thank you so much for replying! We were able to build the select-tf-ops aar but are still getting "Ops not found" in Unity. Do we need to create a new delegate inside Unity to link select-ops with the interpreter? Sorry if this is an obvious question, new to building plugins for Unity.

[asus4]

@rajandeepsingh13 Yes, We need to implement a corresponding FlexDelegate.cs which inherits the IDelegate interface and assigns it into InterpreterOptions.AddDelegate. Possibly the Flex delegate in TFLite may also need to be modified slightly.

[matthew-so]

@asus4 I'm working with @rajandeepsingh13 on this; I'd like to thank you for your help! We'll ask if we have implementation questions.

[matthew-so]

@asus4 How did you figure out which Interpreter and Delegate functions were exposed by TensorFlowLibrary, and where does the Android TensorFlowLibrary (libtensorflowlite_jni) come from?

[asus4]

@rajandeepsingh13 @matthew-so

TFL_CAPI_EXPORT is the keyword to be exposed to the C FFI library. So you can find all exposed methods if you search TFL_CAPI_EXPORT in the TFLite source. After compiling the library, I usually check whether the method is exposed in the library file using a binary reader. You can find the method name if it is exposed correctly. like this ↓

the libtensorflowlite_jni is a part of the tensorflow-lite.aar. you will find the libtensorflowlite_jni for multiply architectures when you unzip the aar.

#270 is the PR for supporting NN Api Delegate. This might help to understand what is required to add a new delegate.

[asus4]

and the PR is welcome once you manage to support it. (although I may separate the repository because the file size of flex delegate will be larger when we support all platforms)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[asus4]

I reopened this issue for the feature development plan. as I received several requests for supporting this within Unity.

Once the flex-delegate library is built and linked to the Unity project, calling this API should work.

https://github.com/tensorflow/tensorflow/blob/4dacf3f368eb7965e9b5c3bbdd5193986081c3b2/tensorflow/lite/core/c/c_api_experimental.h#L231-L252

TFL_CAPI_EXPORT extern TfLiteInterpreter*
TfLiteInterpreterCreateWithSelectedOps(const TfLiteModel* model,
                                       const TfLiteInterpreterOptions* options);

[Ale1]

Hi @asus4 . Thanks for the great plugin. I feel I'm half-way there to using select tf ops, but got stuck in the registration process:

(1) built flex tf lite (ok) (2) added to unity project (ok) (3) am able to expose and use c_api_experimental, including TfliteInterpreterCreateWithSelectedOps (ok)

However, that interpreter creation process differs from the normal one with TfLiteInterpreterCreate because it requires registering all the ops needed, including the built-in ones.
But when I attempt to register ops, I get failed registration errors. I am using the Registration class provided by your package, as well as the builtinOperator enum in your package. But my guess is that Im failing to implement properly because I have to explicitly define the Registration delegates. My current attempt:

/// Modified Intepreter constructor that uses TfLiteInterpreterCreateWithSelectedOps
 public Interpreter(byte[] modelData, InterpreterOptions options)
 {
            modelDataHandle = GCHandle.Alloc(modelData, GCHandleType.Pinned);
            IntPtr modelDataPtr = modelDataHandle.AddrOfPinnedObject();
            model = TfLiteModelCreate(modelDataPtr, modelData.Length);
            if (model == IntPtr.Zero) throw new Exception("Failed to create TensorFlowLite Model");

            this.options = options ?? new InterpreterOptions();

            // Create builtin op 'equal' registration
            var EqualRegistration = new Registration()
            {
                builtinCode = (int) BuiltinOperator.Equal,
                version = 1
                //todo:  probably missing here:
                // initDelegate = ???,
                // freeDelegate = ???,
                // prepareDelegate = ???,
               // invokeDelegate = ???
            };
            
            // include in options register for builtin op 'Equal'
            TfLiteInterpreterOptionsAddBuiltinOp(this.options.nativePtr, (BuiltinOperator)EqualRegistration.builtinCode, ref EqualRegistration,(uint)1,(uint)1);
            
            interpreter = TfLiteInterpreterCreateWithSelectedOps(model, options.nativePtr);
            if (interpreter == IntPtr.Zero) throw new Exception("Failed to create TensorFlowLite Interpreter");
        } 

Unfortunatley, there is very little documentation from tensorflow on how to create those delegates... (see https://www.tensorflow.org/lite/api_docs/c/struct/tf-lite-registration) If you have a working example of a builtin op delegates that would be super helpful to figuring out the rest.

[asus4]

Hi @Ale1, Thanks for sharing your code. I agree that there is no documentation for the C API except for looking into the TFLite source code.

The API in Android TFLite looks like the following. So, probably we can implement a similar API in C# as well. I need to try it out myself for more investigation.

final Interpreter.Options options = new Interpreter.Options();
FlexDelegate flexDelegate = new FlexDelegate();
options.addDelegate(flexDelegate);
final interpreter = new Interpreter(buffer, options);

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.