How to use Video source from outside Unity for person tracking

[shahanhyperspace]

I have a video playing in another software, I am getting the video feed in unity using NDI/Spout. https://github.com/keijiro/KlakNDI

I am currently using the MultiObjectTrackingExample scene where I am just using a pre recorded video. How can I do this in OpenCV? I have the lastet version of OpenCV and doing this on Windows 11.

[EnoxSoftware]

I have created an example that combines KlacNDI and OpenCVForUnity, using AsyncReadback to convert a RenderTexture retrieved from KlacNDI to a Mat.

Setup

1. Import KlacNDI from Package Manager.
2. Import Test Cards from Package Manager.
3. Import OpenCVForUnity.
4. Import KlakNDIWithOpenCVForUnityExample.uniitypackage.

KlakNDIWithOpenCVForUnityExample.zip

[shahanhyperspace]

For reference to anyone looking into a solution like this, here is the test cards link : https://github.com/keijiro/TestCards

[shahanhyperspace]

This works like a charm. I am getting the camera feed from outside unity. The camera feed has people walking.

How can I input the camera feed form the quad to the MultiObjectTrackingExample scene?

[EnoxSoftware]

I wanted to try something like this, so I created a multi-object tracking example.

KlakNDIWithOpenCVForUnityExample_v0.2.zip

In this screenshot, LiveCamera feeds is acquired. You may need to adjust the frame skip value to display the feed smoothly.

Youtube LiveCamera ➡️ VLC with NDI Plugin ➡️ klacNDI

[shahanhyperspace]

@EnoxSoftware works perfectly. I am able to merge seven camera inputs and then send it to openCV for person detection.

Quick question, it sometimes detects objects in the video as well. I did edit coco.names and have only person in it but it still detects objects. any solution to this?

I see there is a label on the object saying "person" how can I find where is this person coming from? associated with the ID

[EnoxSoftware]

To do so, you will need to create a new YOLOX model for object detection that classifies only “person”. Or you can simply add a conditional branch that ignores IDs other than “person” to the process of processing the inference results of the model you are currently using.

[shahanhyperspace]

How and where would i add the conditional branch?

[EnoxSoftware]

I created a simple visualize method, with a branch from results that bypasses the process if classId is non-zero. Please add a scene to your project and try it out. ObjectDetectionYOLOExample_Person.zip

        ///////////////////////////////////////////////////////////////////////////////////////////////////////
        public virtual void visualize_Person(Mat image, Mat results, bool isRGB = false)
        {
            if (image.IsDisposed)
                return;
            if (results.empty() || results.cols() < 6)
                return;

            DetectionData[] data = objectDetector.getData(results);

            foreach (var d in data.Reverse())
            {
                float left = d.x1;
                float top = d.y1;
                float right = d.x2;
                float bottom = d.y2;
                float conf = d.conf;
                int classId = (int)d.cls;

                // Ignore if classID is not 0 (person).
                if (classId != 0)
                    continue;

                // ID:0(person)
                Scalar c = new Scalar(255, 56, 56, 255);
                Scalar color = isRGB ? c : new Scalar(c.val[2], c.val[1], c.val[0], c.val[3]);

                Imgproc.rectangle(image, new Point(left, top), new Point(right, bottom), color, 2);

                string label = $"{"person"}, {conf:F2}";

                int[] baseLine = new int[1];
                Size labelSize = Imgproc.getTextSize(label, Imgproc.FONT_HERSHEY_SIMPLEX, 0.5, 1, baseLine);

                top = Mathf.Max((float)top, (float)labelSize.height);
                Imgproc.rectangle(image, new Point(left, top - labelSize.height),
                    new Point(left + labelSize.width, top + baseLine[0]), color, Core.FILLED);
                Imgproc.putText(image, label, new Point(left, top), Imgproc.FONT_HERSHEY_SIMPLEX, 0.5, Scalar.all(255), 1, Imgproc.LINE_AA);
            }
        }
        ///////////////////////////////////////////////////////////////////////////////////////////////////////