[BUG]: "Exception: There are more object colors than there are objects" using HDRI

[mikkelmedm]

Describe the bug Duplicate of #283. This error seem to pop up after a few runs at a certain angle, where it appears camera is facing the sky. Seem the fix of #283 is already implemented in the newer versions, so very curios to what could help in this scenario.

General Information

1. Which BlenderProc version are you using? Newest version (2.2.0)
2. On which operating system are you? Linux
3. Have you checked the issue tracker to see if a similar issue has been opened? Yes.
4. Have you changed BlenderProc in any way besides the config file? If yes, are you sure that this change does not affect the problem you are having? No. To Reproduce Steps to reproduce the behavior:
5. Provide the full command you used to run BlenderProc: blenderproc run main.py output
6. Provide the full python file, you used:

import blenderproc as bproc
import argparse
import os
import numpy as np
from numpy.random import choice
import random
import bpy
import importlib
from mathutils import Vector
from blenderproc.python.types.MeshObjectUtility import MeshObject
import time

parser = argparse.ArgumentParser()
# Paths:
parser.add_argument('output_dir', default="trackar_data/output", help="Path to where the final files will be saved ")
parser.add_argument('--cc_textures_path', default="resources/cctextures", help="Path to downloaded cc textures")

# Parameters: 
parser.add_argument('--bound_box', default=False, help="True for BBox keypoints - False for unique keypoints")

parser.add_argument('--scene', default="heineken.blend", help="Path to the scene.blend file")
parser.add_argument('--object_name', default="beer", help="Name of object in .blend file with keypoints")
parser.add_argument('--multi_obj', default=False, help="If the scene contains multiple objects to load one at random")

parser.add_argument('haven_path', nargs='?', default="resources/haven_dataset", help="The folder where the `hdri` folder can be found, to load an world environment")

parser.add_argument('--num_runs', default=10, help="Number of runs")
parser.add_argument('--frames_per_run', default=15, help="Number of frames for each run")
parser.add_argument('--out_size', default=(256, 512), help="Output width and output height")
parser.add_argument('--cam_dist_min', default=0.25, help="Minimum radius of camera sampler shell")
parser.add_argument('--cam_dist_max', default=0.8, help="Maximum radius of camera sampler shell")
parser.add_argument('--distractor_objs_min', default=2, help="Minimum number of occluding/distractor objects to add to the scene")
parser.add_argument('--distractor_objs_max', default=15, help="Maximum number of occluding/distractor objects to add to the scene")
args = parser.parse_args()

bproc.init()

output_dir = "/".join(args.output_dir.split("/")[:-1]) if args.output_dir[-1].isdigit() else args.output_dir

# Set to True for faster outputs during debbuging
DEBUG = True


if DEBUG:
    bproc.renderer.set_max_amount_of_samples(10)
    
else:
    distractor_objs_max = args.distractor_objs_max
    distractor_objs_min = args.distractor_objs_min
    bproc.renderer.set_light_bounces(  
                            diffuse_bounces = 200, 
                            glossy_bounces = 200,
                            max_bounces = 200,
                            transmission_bounces = 200, 
                            transparent_max_bounces = 200
                            )
    bproc.renderer.set_max_amount_of_samples(256) # TODO: Should we leave this as default on 1024 ?? takes longer
          

# load the blender object(s) into the scene
load_blend_objs = bproc.loader.load_blend(args.scene)

scale = 0.15

# set segmap_class (category_id) and physics properties 
for obj in load_blend_objs:
    obj.set_cp("category_id", 1)
    
    if not args.multi_obj and obj.get_name() == args.object_name:
        obj_of_interest = obj 
        object_name = obj_of_interest.get_name()
        obj.set_scale(Vector((scale, scale, scale)))

# Load subset of cctextures each rerun:
cc_textures = random.sample(bproc.loader.load_ccmaterials(args.cc_textures_path), 50)

light_point = bproc.types.Light()
light_point.set_energy(200)

# Iteratively generate a new scene with random camera/light/object placements:        
for r in range(args.num_runs):

    # If .blend file contains more objects, select one at random, hide the others:
    if args.multi_obj:
        obj_of_interest = np.random.choice(load_blend_objs)
        object_name = obj_of_interest.get_name()
        obj_of_interest.set_scale(Vector((scale, scale, scale)))
        for obj in load_blend_objs:
            obj.hide(True)
        obj_of_interest.hide(False)

    bproc.utility.reset_keyframes()
    FOV = np.random.uniform(1.04, 1.277) 
    bproc.camera.set_intrinsics_from_blender_params(FOV, args.out_size[0], args.out_size[1], lens_unit="FOV")

    
    # Collect all materials
    materials = bproc.material.collect_all()

    # Set a random hdri from the given haven directory as background
    haven_hdri_path = bproc.loader.get_random_world_background_hdr_img_path_from_haven(args.haven_path)
    bproc.world.set_world_background_hdr_img(haven_hdri_path)

    room_planes = [bproc.object.create_primitive('PLANE', scale=[2, 2, 1])]
    
    # sample light color and strength from ceiling
    light_plane = bproc.object.create_primitive('PLANE', scale=[3, 3, 1], location=[0, 0, 10])
    light_plane.set_name('light_plane')
    light_plane_material = bproc.material.create('light_material')
    light_plane_material.make_emissive(emission_strength=np.random.uniform(3,6), replace=True,
                                    emission_color=np.random.uniform([0.5, 0.5, 0.5, 1.0], [1.0, 1.0, 1.0, 1.0]))    
    light_plane.replace_materials(light_plane_material)

    # sample point light on shell
    
    light_point.set_color(np.random.uniform([0.5, 0.5, 0.5], [1, 1, 1]))
    location = bproc.sampler.shell(center = [0, 0, 0], radius_min = 1, radius_max = 1.5,
                            elevation_min = 5, elevation_max = 89)
    light_point.set_location(location)

    # sample CC Texture and assign to room planes
    random_cc_texture = np.random.choice(cc_textures)
    for idx, plane in enumerate(room_planes):
        plane.replace_materials(random_cc_texture)
    
    # Define a function that samples the initial pose of a given object above the ground 
    def sample_initial_pose(obj: bproc.types.MeshObject):
        obj.set_location(bproc.sampler.upper_region(objects_to_sample_on=room_planes[0:1],
                                                    min_height=1, max_height=4, face_sample_range=[0.4, 0.6]))
        obj.set_rotation_euler(np.random.uniform([0, 0, 0], [0, 0, np.pi * 2]))

    # Sample objects on the given surface
    placed_objects = bproc.object.sample_poses_on_surface(objects_to_sample=[obj_of_interest],
                                                surface=room_planes[0],
                                                sample_pose_func=sample_initial_pose,
                                                min_distance=0.01,
                                                max_distance=1)

    # BVH tree used for camera obstacle checks
    bvh_tree = bproc.object.create_bvh_tree_multi_objects(placed_objects)

    
    poses = 0
    while poses < args.frames_per_run:
        # Sample location
        location = bproc.sampler.shell(center = obj_of_interest.get_location(),
                                radius_min = args.cam_dist_min,
                                radius_max = args.cam_dist_max,
                                elevation_min = 10,
                                elevation_max = 45)
        
        # Determine point of interest in scene
        poi = bproc.object.compute_poi([obj_of_interest])
        
        # Add variation to the poi (so camera is not always statically pointing at obj_of_interest)
        vari = [0,0,0]
        vari[random.randint(0, len(vari)-1)] = random.uniform(0, 0.1)
        poi += vari

        # Compute rotation based on vector going from location towards poi
        rotation_matrix = bproc.camera.rotation_from_forward_vec(poi - location, inplane_rot=np.random.uniform(-0.7854, 0.7854))
        # Add homog cam pose based on location an rotation
        cam2world_matrix = bproc.math.build_transformation_mat(location, rotation_matrix)
        
        # Check that obstacles are at least 0.2 meter away from the camera and make sure the view interesting enough
        if bproc.camera.perform_obstacle_in_view_check(cam2world_matrix, {"min": 0.2}, bvh_tree):
            # Persist camera pose
            bproc.camera.add_camera_pose(cam2world_matrix)
            poses += 1


    # render the whole pipeline
    data = bproc.renderer.render()
    # Render segmentation masks (per class and per instance)
    data.update(bproc.renderer.render_segmap(map_by=["class"]))
    # Write data in custom format -> "blenderproc/python/writer/CustomWriter.py"
    bproc.writer.write_custom(os.path.join(output_dir), data, object_name, args.bound_box, append_to_existing_output=True)

3. Provide a link to all 3D models you used, if they are from one of the publicly available supported datasets, provide the name or path so that it is possible to reproduce the error. https://drive.google.com/file/d/1piSHdvrguewaVdPGTkY5O_8rrdmlpByq/view?usp=sharing

[mikkelmedm]

Okay, so guess I found a workable solution. But not sure how good it is. In utility/SegMapRendererUtility.py I changed it to

def _set_world_background_color(color):
        """ Set the background color of the blender world obejct.

        :param color: A 3-dim array containing the background color in range [0, 255]
        """
        nodes = bpy.context.scene.world.node_tree.nodes
        links = bpy.context.scene.world.node_tree.links

        # Unlink any incoming link 
        if len(nodes.get("Background").inputs['Color'].links) > 0:
            links.remove(nodes.get("Background").inputs['Color'].links[0])
        # Set strength to 1 as it would act as a multiplier
        nodes.get("Background").inputs['Strength'].default_value = 0 # CHANGED THIS TO 0 INSTEAD OF 1
        nodes.get("Background").inputs['Color'].default_value = color + [0] # CHANGED THIS TO 0 INSTEAD OF 1

This seem to avoid the error, but not knowledable enough in Blender to know the consequences.

[mikkelmedm]

Above didn't help. Still working on a fix.

[mikkelmedm]

Still looking for a fix to this

[saprrow]

Same issue, hope for solution！ T^T。

[cornerfarmer]

Hey @mikkelmedm and @saprrow,

sorry for the late response. Apparently the hdr image was still active during rendering the seg maps which lead to having more colors than objects in the image. You can fix the error by adding two lines to the end of _set_world_background_color:

def _set_world_background_color(color: mathutils.Vector):
    """ Set the background color of the blender world object.

    :param color: A 3-dim array containing the background color in range [0, 255]
    """
    nodes = bpy.context.scene.world.node_tree.nodes
    links = bpy.context.scene.world.node_tree.links

    # Unlink any incoming link that would overwrite the default value
    if len(nodes.get("Background").inputs['Color'].links) > 0:
        links.remove(nodes.get("Background").inputs['Color'].links[0])
    # Set strength to 1 as it would act as a multiplier
    nodes.get("Background").inputs['Strength'].default_value = 1
    nodes.get("Background").inputs['Color'].default_value = color + [1]

    # Make sure the background node is connected to the output node
    output_node = Utility.get_the_one_node_with_type(nodes, "Output")
    links.new(nodes.get("Background").outputs["Background"], output_node.inputs["Surface"])

I will create a PR to fix this on the main branch