Performance over grayscale/Infrared input

[lghasemzadeh]

Hello,

Do the tensorflow face landmarks detection and Iris landmarks models work on Infrared/grayscal image/video? I have a camera which captures Infrared streams, can I run this model over the infrared videos that I recorded with that camera previously? does the model work the same as when it works with RGB videos or real-time RGB streams. @tensorflow @tensorflow-models/face-landmarks-detection @tensorflow-models/facemesh here is the link to the library: https://blog.tensorflow.org/2020/11/iris-landmark-tracking-in-browser-with-MediaPipe-and-TensorFlowJS.html

Actually I want to apply the model/algorithm over infrared/grayscale stream real-time.

Thank

[sgowroji]

Hi @lghasemzadeh, We would like to hear it from you. We have not tried as mentioned above. Please share your observation.

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[sgowroji]

Closing this ticket as we don't hear any update from the reporter!

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No

[lghasemzadeh]

Hello,

I have tested MediaPipe FaceMesh on infrared streams, and it works. The performance is not as perfect as with RGB, but it functions without major issues. Infrared image is a type of grayscale image and it has only one channel

1. Now the problem is that I want to understand how FaceMesh can work on an image with 1-channel, while it is trained on RGB (3-channel) input.
2. During the pre-processing steps or training, is there any step where RGB is converted into grayscale? I did some research on this topic a while ago, but unfortunately, I cannot find those resources anymore.
3. would you please share links to the papers/models/details of the facemesh solution to figure out this behavior?

I am waiting for your response.

Thank you.

[rohitgarud]

@lghasemzadeh, Maybe stacking the single-channel matrix three times to create 3-channel image might help. Which camera are you using to capture IR stream?

[lghasemzadeh]

Hello @sgowroji, I am waiting for your responce.

@rohitgarud there is no problem in fitting, the library works, but I want to know why?

[rohitgarud]

@lghasemzadeh as you mentioned the performance with the single-channel images is not as good as RGB, maybe performance can improve by staking the single channel to create something like pseudo-RGB.. I know color channels don't work like that but it might improve the performance.. I think the amount of monochrome data used for training (if that's the case) will be much less than RGB data.. let's wait for @sgowroji to give a definitive answer

[khengari77]

@lghasemzadeh

Infrared image is a type of grayscale image and it has only one channel

I doubt that infrared has one channel. It should have 3. I ran into the same problem yesterday and after reading the SolutionBase.py I found that it actually checks for the number of channels and if it's not 3 it will raise an exception. I think that the camera you are using outputs 3 gray channels that are equal to each other giving the effect of 1 grayscale channel. Nonetheless I have an infrared camera lying around and I will try to test it as soon as possible.

[jianyao05]

@lghasemzadeh

Infrared image is a type of grayscale image and it has only one channel

I doubt that infrared has one channel. It should have 3. I ran into the same problem yesterday and after reading the SolutionBase.py I found that it actually checks for the number of channels and if it's not 3 it will raise an exception. I think that the camera you are using outputs 3 gray channels that are equal to each other giving the effect of 1 grayscale channel. Nonetheless I have an infrared camera lying around and I will try to test it as soon as possible.

Hey ! currently planning to start a new project and wondering if there was any update on your end? Thankyouu

[khengari77]

@lghasemzadeh

Infrared image is a type of grayscale image and it has only one channel

I doubt that infrared has one channel. It should have 3. I ran into the same problem yesterday and after reading the SolutionBase.py I found that it actually checks for the number of channels and if it's not 3 it will raise an exception. I think that the camera you are using outputs 3 gray channels that are equal to each other giving the effect of 1 grayscale channel. Nonetheless I have an infrared camera lying around and I will try to test it as soon as possible.

Hey ! currently planning to start a new project and wondering if there was any update on your end? Thankyouu

Unfortunately no. I found that the performance degrades when you use grayscale. There are two theoretical solutions for this problem. Either you train your own model that extracts the landmarks you need or you use a colorizing model and then feed the output to mediapipe.

[jianyao05]

@lghasemzadeh

Infrared image is a type of grayscale image and it has only one channel

I doubt that infrared has one channel. It should have 3. I ran into the same problem yesterday and after reading the SolutionBase.py I found that it actually checks for the number of channels and if it's not 3 it will raise an exception. I think that the camera you are using outputs 3 gray channels that are equal to each other giving the effect of 1 grayscale channel. Nonetheless I have an infrared camera lying around and I will try to test it as soon as possible.

Hey ! currently planning to start a new project and wondering if there was any update on your end? Thankyouu

Unfortunately no. I found that the performance degrades when you use grayscale. There are two theoretical solutions for this problem. Either you train your own model that extracts the landmarks you need or you use a colorizing model and then feed the output to mediapipe.

oh... are you able to provide some insights or opinions on my project?

i plan to be able to detect leg movements under a blanket and currently im thinking of using either thermal or infrared cameras to be able to detect. im not sure if mediapipe or other models are able to do it.

thoughts and alternatives i had thought about was to use a heat map/color map to change it to rgb color scale but im not too sure

do share your thoughts or opinions on this thankyouuu

[khengari77]

@lghasemzadeh

Infrared image is a type of grayscale image and it has only one channel

I doubt that infrared has one channel. It should have 3. I ran into the same problem yesterday and after reading the SolutionBase.py I found that it actually checks for the number of channels and if it's not 3 it will raise an exception. I think that the camera you are using outputs 3 gray channels that are equal to each other giving the effect of 1 grayscale channel. Nonetheless I have an infrared camera lying around and I will try to test it as soon as possible.

Hey ! currently planning to start a new project and wondering if there was any update on your end? Thankyouu

Unfortunately no. I found that the performance degrades when you use grayscale. There are two theoretical solutions for this problem. Either you train your own model that extracts the landmarks you need or you use a colorizing model and then feed the output to mediapipe.

oh... are you able to provide some insights or opinions on my project?

i plan to be able to detect leg movements under a blanket and currently im thinking of using either thermal or infrared cameras to be able to detect. im not sure if mediapipe or other models are able to do it.

thoughts and alternatives i had thought about was to use a heat map/color map to change it to rgb color scale but im not too sure

do share your thoughts or opinions on this thankyouuu

I am not sure but I think that this is achievable even without ml. You can research movement detection. It's an image processing technique that detects changes in pixels.

[kalyan2789g]

Hi @lghasemzadeh,

Thank you for reporting this issue. This issue is related to the legacy feature that is no longer actively supported and support has been completely ended.

We recommend migrating to the new MediaPipe API’s for continued support and access to the most recent documentation to new MediaPipe API’s here. Please consult the release notes for information on the latest updates.

If you encounter any issues with the new MediaPipe API’s, please open a new GitHub issue with detailed debugging information to facilitate our investigation.

Please follow the release notes to stay up to date with the latest developments which are happening in the MediaPipe space. MediaPipe

Thanks, @kalyan2789g

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[github-actions[bot]]

This issue was closed due to lack of activity after being marked stale for past 7 days.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No