Get labels and locations instead of rendering objects on screen

[Topstar88]

Hi @nihui , This repo works very well. Is there any way to get label texts and locations instead of rendering objects on screen? I need to get objects' information to use them for the next processing. Best Regards.

[cyrillkuettel]

Hi @Topstar88 I assume you want the objects information on the Java layer.

Yes it's possible. Just take a look at this fork. The important part is here:

for (auto &object : objects) {
      strcat(str, class_names[object.label]);
   }
   jstring js = string2jstring(ncnn_env, str);
   ncnn_env->CallVoidMethod(ncnn_thiz, ncnn_callback, js); // calls java method

The locations is also possible. You can get the locations (x/y) like this

    for (size_t i = 0; i < objects.size(); i++)
    {
        const Object& obj = objects[i];
        int x = obj.rect.x;
        int y = obj.rect.y 
      // do something with x and y
   }

[Houangnt]

Hi @Topstar88 I assume you want the objects information on the Java layer.

Yes it's possible. Just take a look at this fork. The important part is here:

for (auto &object : objects) {
      strcat(str, class_names[object.label]);
   }
   jstring js = string2jstring(ncnn_env, str);
   ncnn_env->CallVoidMethod(ncnn_thiz, ncnn_callback, js); // calls java method

The locations is also possible. You can get the locations (x/y) like this

    for (size_t i = 0; i < objects.size(); i++)
    {
        const Object& obj = objects[i];
        int x = obj.rect.x;
        int y = obj.rect.y 
      // do something with x and y
   }

I tried the above source code but got this error. Do you know fix it ?

[cyrillkuettel]

It's hard to say from the screenshot. @Houangnt Can you provide the link to the project then I can take a look at it.

[Houangnt]

It's hard to say from the screenshot. @Houangnt Can you provide the link to the project then I can take a look at it.

https://drive.google.com/file/d/1lUxGFU4a71FFlAnIt-36PT_Ex0hbsL-s/view?usp=sharing. This is my project, you can download and build it. many thanks

[cyrillkuettel]

There you go my friend.

It works. Find the project here. You can download it as zip or clone it with git. It has all dependencies (opencv-mobile and ncnn) already included.

The locations (rectangles) I have not yet implemented, but now it's easy to do. Just change the method to accept more parameters. Let me know if you have any more questions :)

[cyrillkuettel]

@Houangnt I'm happy to help. Let me know if you encounter any problems.

[Houangnt]

@Houangnt I'm happy to help. Let me know if you encounter any problems.

Do you know how to increase the resolution of this app?. Because after I have the coordinates I will crop and save the image, but the resolution of the image is very low :(((

[cyrillkuettel]

There are a couple of options:

• To get the highest resolution possible, you could take a picture with the Camera first, then save the image and crop the image from that image. If you go down that path, I'd advise to use CameraX However, it is not trivial to control the camera with Android.

• Another option I know that works PixelCopy to copy the content of SurfaceView. That's probably simpler to implement. For this to work, you need to request storage permission for android.

You might find this link helpful https://stackoverflow.com/questions/25086263/take-screenshot-of-surfaceview

[Houangnt]

There are a couple of options:

• To get the highest resolution possible, you could take a picture with the Camera first, then save the image and crop the image from that image. If you go down that path, I'd advise to use CameraX However, it is not trivial to control the camera with Android.
• Another option I know that works PixelCopy to copy the content of SurfaceView. That's probably simpler to implement. For this to work, you need to request storage permission for android.

You might find this link helpful https://stackoverflow.com/questions/25086263/take-screenshot-of-surfaceview many tks, i will try it

[cyrillkuettel]

Hi Did it work? i might have time tomorrow to develop something

[Houangnt]

Hi Did it work? i might have time tomorrow to develop something

i tried it with copypixel but i think copy bitmap via surface app is not reach best resolution. You can easily see the image via this app is not have a good resolution. This is my result :

[cyrillkuettel]

It might be possible to maximize the SurfaceView resolution by setting some parameters in Ndk Camera. I actually don't know, I've never tried it. The documentation on Ndk Camera is quite sparse.

You can modify the resoution of the image captured from the camera, but I don't know if this affects the preview size. It's possible like this:

https://github.com/nihui/ncnn-android-nanodet/blob/acb9dd3cbeb3b5e090c6baea73bba501b66cd2de/app/src/main/jni/ndkcamera.cpp#L183

By the way, can you create a repository of the project? That way i can help better.

[Topstar88]

There you go my friend.

It works. Find the project here. You can download it as zip or clone it with git. It has all dependencies (opencv-mobile and ncnn) already included.

The locations (rectangles) I have not yet implemented, but now it's easy to do. Just change the method to accept more parameters. Let me know if you have any more questions :)

@cyrillkuettel, It worked well. Thank you. And I have another request. Your new repository can detect face masks as well as glasses. Could we make it possible to detect face masks, glasses, and other objects as well? and is there any way to distinguish between common glasses and sunglasses(black glasses)? Thanks in advance.

[cyrillkuettel]

You can load two ncnn models, one for masks, glasses and one for other objects. Then you make inference with both models sequentially (One after the other). But this would be not very efficient, there will be a drop in fps probably. What kind of other objects are you interested in?

It's not possible to distinguish between common glasses and sunglasses. The model you're currently using (yolov5-lite) can only detect a generic type of glasses. You would need to find a model which can do that, or train your own model.

[Houangnt]

It might be possible to maximize the SurfaceView resolution by setting some parameters in Ndk Camera. I actually don't know, I've never tried it. The documentation on Ndk Camera is quite sparse.

You can modify the resoution of the image captured from the camera, but I don't know if this affects the preview size. It's possible like this:

https://github.com/nihui/ncnn-android-nanodet/blob/acb9dd3cbeb3b5e090c6baea73bba501b66cd2de/app/src/main/jni/ndkcamera.cpp#L183

By the way, can you create a repository of the project? That way i can help better.

i improved resolution of image . I try to get coordinate of bbox but sometimes i crop true or i crop only 1/2 of bbox, maybe the coordinate return not absolute exact and i dont find the reason of this problem

[cyrillkuettel]

Unfortunately, I can't solve your problem. I need to see the code.