How can I tell if an object is obscured by another object in the camera view?

[ccteaher]

Describe the issue

Hi, I need to make a coco data:

if the object is occlusioned, don't render its mask.

But I can't tell if it's obscured by other objects in the camera's view.

How can I tell if an object is obscured by another object in the camera view?

Look forward to your reply very much, thank you.

Minimal code example

No response

Files required to run the code

No response

Expected behavior

How to tell if the object is occlusioned from the camera Angle?

BlenderProc version

3.3.1

[cornerfarmer]

Hey @cuijiale123,

one thing you can do is: Render for each object a separate segmentation mask where all other objects are hidden. Then in the end you check whether the object mask is bigger when rendering the object alone compared to rendering it with all other objects together. If that is the case, the object is occluded by another object.

You can also apply the same logic using raycasts. This is probably more efficient but might be harder to implement

[tlpss]

stumbled upon this issue and quickly wanted to add my 50 cents:

I think @cornerfarmer is right that the easiest way to determine object visibility is to render segmentation masks and compare them.

If you want to do a more fine-grained visibility check, you can use raycasting. I've done this recently to check if a certain vertex is visible:

def is_vertex_occluded_for_scene_camera(co: Vector, helper_cube_scale: float = 0.0001) -> bool:
    """Checks if a vertex is occluded by objects in the scene w.r.t. the camera.

    Args:
        co (Vector): the world space x, y and z coordinates of the vertex.

    Returns:
        boolean: visibility
    """
    co = Vector(co)

    bpy.context.view_layer.update()  # ensures camera matrix is up to date
    scene = bpy.context.scene
    camera_obj = scene.camera  # bpy.types.Object

    # add small cube around coord to make sure the ray will intersect
    # as the ray_cast is not always accurate
    # cf https://blender.stackexchange.com/a/87755
    bpy.ops.mesh.primitive_cube_add(location=co, scale=(helper_cube_scale, helper_cube_scale, helper_cube_scale))
    cube = bpy.context.object
    direction = co - camera_obj.location
    hit, location, _, _, _, _ = scene.ray_cast(
        bpy.context.view_layer.depsgraph,
        origin=camera_obj.location + direction * 0.0001,  # avoid self intersection
        direction=direction,
    )

    if DEBUG:
        print(f"hit location: {location}")
        bpy.ops.mesh.primitive_ico_sphere_add(
            location=location, scale=(helper_cube_scale, helper_cube_scale, helper_cube_scale)
        )

    # remove the auxiliary cube
    if not DEBUG:
        bpy.data.objects.remove(cube, do_unlink=True)

    if not hit:
        raise ValueError("No hit found, this should not happen as the ray should always hit the vertex itself.")
    # if the hit is the vertex itself, it is not occluded
    if (location - co).length < helper_cube_scale * 2:
        return False
    return True

You could also use this to check if an object is visible by iterating over all vertices as below. Though I am not sure how efficient this is, depending on your mesh resolution and image resolution it might be better to cast rays from the camera instead.

def is_object_occluded_for_scene_camera(obj: bpy.types.Object) -> bool: """Checks if all vertices of an object are occluded by objects in the scene w.r.t. the camera.

Args:
    obj (bpy.types.Object): the object.

Returns:
    boolean: visibility
"""
for vertex in obj.data.vertices:
    coords = obj.matrix_world @ vertex.co
    if not is_vertex_occluded_for_scene_camera(coords):
        return False
return True