Doing Image Processing on the RGB Image

[KansaiTraining]

I have read the examples and I can see that I will have the image loaded on _latestImageBytes. Now, same as what SoylentGraham commented in https://github.com/VulcanTechnologies/HoloLensCameraStream/issues/10 I can understand of course that this is an 1D array. I suppose the BGRA is similar to the Format32bppArgb format.

I want to do some Image Processing on this image. (I have already the algorithms I want to implement so no problem there) Can anyone give some advice or pointers on how to proceed?

---

In the file MatrixUsageApp.cs or in VideoPanelApp.cs I see there is a OnFrameSampleAcquired function. I get that this is the function that gets called on every frame. Inside the function there is the UnityEngine.WSA.Application.InvokeOnAppThread call .

I don't understand this so well. Is this being called on another thread? Should I put all my processing inside this? (I do a lot of processing) . And if it is done on another thread, how can I finally get some results back to the Unity UI Thread (that I suppose is the one starting it)

Thousand thanks for any help. I really like this library.

[motionsmith]

Yes, you probably want to do your work within the InvokeOnAppThread callback. Here's why:

OnFrameSampleAcquired does not get called on Unity's thread, so InvokeOnAppThread is a way of returning your execution stack to Unity's thread. The reason why we want to return to Unity's thread is because calls to Unity's API will fail if they are called from a non-Unity thread.

However, you don't NEED to call InvokeOnAppThread if you won't be making any Unity API calls. However, I'm not sure if it's a good idea to do your processing on the same thread as the CameraStream library anyway. If you're doing expensive CPU-processing of the image bytes then it's probably best to do them on an independent thread anyway.

[KansaiTraining]

Thank you very much for your reply. I have been thinking about what you wrote, trying to understand it fully and I have come to a follow up question: Would it be a good idea (or good design) to do something like this:

void OnFrameSampleAcquired(VideoCaptureSample sample)
{  //the usual code here

//Now I have the data on _latestImageBytes

ProcessAsynchronously(_latestImageBytes);

UnityEngine.WSA.Application.InvokeOnAppThread( //....the rest here

);
}

void  async void ProcessAsynchronously(byte[] image)
{
await Process1(image);
await Process2(image);
//....
}

Or maybe call the InvokeOnAppThread inside the ProcessAsynchronously function and not in OnFrameSampleAcquired

The idea would be to do the processing in a asynchronous process. I would like to call an await inside the OnFrameSampleAcquired but I don't think that is possible. So I put it inside the function. I have not figure it out how to return any results, other than modify the byte[] array but something tells me this is not a good strategy.

Other concern I have (related to the previous) is that since the FrameSampleAcquiredCallback gets called every frame, it will rewrite the _latestImageBytes every frame (sample.CopyRawImageDataIntoBuffer) and if my processing last too long it won't go well.

I read that instead of using this callback we can use RequestNextFrameSample() but I don't know if that is the correct way to do it or how to actually do it correctly.

Any recommendation or insight will be greatly appreciated

[motionsmith]

Once you copy the raw bytes into your own byte[], I think you can work with it in your own thread if you'd like. But yes, if you're doing processing on every frame you might run into trouble, depending on how much processing you're doing, thread priority, etc. If your processing is thread safe, there could be a backup of threads waiting for each other to unlock. I'm not an expert in video processing pipelines, so I'm not sure about the specifics.

[SoylentGraham]

The big thing about bigdata/video processing it balancing/scheduling the work. Long term, you may want a system to decide when to skip data. Buffering up a big todo list (ie. a list of frames) helps with this.

Your approach SEEMS good, but there's a fatal flaw; await/async stuff in c# runs on the SAME thread as it's called from. (This is a big problem for networking on UWP/hololens as all their socket stuff is async/await) It's just interleaved with the other things the thread is doing. So you're putting all your processing on the videofeed's output thread. This is bad! (slows down the camera essentially) If your processing doesn't need to use unity objects (textures, meshes, gameobjects etc) then it should be pretty safe to put on it's own thread (usual multithreading caveats apply). For UWP/Hololens you're pretty much limited to the threadpool, so use something like

System.Action Processing = () =>
{
 Process1( image );
 Process2(image);
};
System.Threading.ThreadPool.Enque( Processing );

This keeps it off the main thread (so doesn't slow down rendering/game logic) and doesn't slow down the camera thread. The internal threadpool doesn't have a huge amount of control, so again, scheduling/balancing you may want to do at a high level. (ie, not have too many threaded actions running/queued at once)

In my own example, I take each frame, add to a queue of frames to process and I have about 2 threads running at once encoding each frame to H264 and then that output goes to another queue for a network thread to send the packets out.

[Upd4ting]

Hey @SoylentGraham I would know if your system of thread that encode in h264 the pictures and then output it in an another queue has been posted on github somewhere? I am really interested to see how h264 encoding work because I may need to do something similar

[SoylentGraham]

I don't have an open source implementation I'm afraid, just the jpeg one is open sourced. Implemented with MediaFoundation. But I may get it open sourced one day as a general cross platform unity h264-encoder. (Currently being used for commercial things, so only available for cash)

[Upd4ting]

Okay no problem thanks anyways :) May I ask if you could help here? https://stackoverflow.com/questions/49075879/opencv-unity-get-3d-position-from-2d-image-position Like you're using HololensCameraStream you may have done something similar