fix(loader): Preserves collection hierarchies when importing them from blend file

[cornerfarmer]

When using

bproc.loader.load_blend("my.blend", data_blocks=["objects", "collections"])

the hierarchies between collections and objects should now be preserved.

@johan-apes I think I found an easier way to fix the collection hierarchies. Could you please check whether this fix also works for you?

[cornerfarmer]

BlenderProcHelper result for run 616:

Done with: [amass_human_poses_python 30s, auto_shading_python 9s, basic_python 15s, blenderkit_python 20s, bop_object_on_surface_sampling_python 100s, bop_object_physics_positioning_python 129s, bop_object_pose_sampling_python 108s, bop_scene_replication_python 181s, camera_depth_of_field_python 19s, camera_object_pose_python 22s, camera_random_trajectories_python 60s, camera_sampling_python 22s, coco_annotations_python 13s, diffuse_color_image_python 21s, dust_python 33s, entity_displacement_modifier_python 15s, entity_manipulation_python 10s, front_3d_object_sampling_python 97s, front_3d_python 102s, front_3d_with_improved_mat_python 113s, git_log 1s, haven_python 26s, ikea_python 18s, kinect_azure_noise_python 13s, lens_distortion_python 17s, lens_distortion_python 12s, light_sampling_python 10s, material_manipulation_python 10s, material_randomizer_python 12s, matterport3d_python 66s, motion_blur_rolling_shutter_python 30s, motion_blur_rolling_shutter_python 28s, multi_render_python 228s, nocs_python 19s, object_pose_sampling_python 11s, on_surface_object_sampling_python 11s, optical_flow_python 20s, physics_convex_decomposition_python 136s, physics_positioning_python 25s, pix3d_python 18s, point_clouds_python 25s, random_backgrounds_python 19s, random_room_constructor_python 80s, replica_python 223s, scenenet_python 49s, scenenet_with_cctextures_python 63s, semantic_segmentation_python 14s, shapenet_python 20s, shapenet_with_scenenet_python 56s, shapenet_with_suncg_python 55s, spotlight_python 23s, stereo_matching_python 48s, stereo_matching_with_projector_python 146s, suncg_basic_python 34s, suncg_with_cam_sampling_python 58s, suncg_with_improved_mat_python 75s, suncg_with_object_replacer_python 61s, tests_python 30s, urdf_loading_and_manipulation_python 61s] Done with 59 of 59 tests. There were errors: :x:
Unit tests output:

................
----------------------------------------------------------------------
Ran 16 tests in 23.972s

OK

urdf_loading_and_manipulation (main_python)

The output of urdf_loading_and_manipulation:main_python:000000_scene_gt_info_json has changed:

Expand

+ root['0'][2]['bbox_visib'][3]: 25
- root['0'][2]['bbox_visib'][3]: 23
+ root['1'][6]['bbox_visib'][0]: 158
- root['1'][6]['bbox_visib'][0]: 157
+ root['1'][6]['bbox_visib'][2]: 22
- root['1'][6]['bbox_visib'][2]: 23

stereo_matching (main_python)

The output of stereo_matching:main_python:disparity has changed: Diff: 0.0314, Pixel diff: 0.0031

Compare image	Newly generated image

Pylint output

Expand

Messages

Module blenderproc (blenderproc/__init__.py)

Line	Col.	Type	Symbol	ID	Obj	Message
12	44	convention	trailing-whitespace	C0303		Trailing whitespace

Module blenderproc.python.camera.CameraProjection (blenderproc/python/camera/CameraProjection.py)

Line	Col.	Type	Symbol	ID	Obj	Message
4	0	warning	unused-import	W0611		Unused create_primitive imported from blenderproc.python.types.MeshObjectUtility
10	0	convention	ungrouped-imports	C0412		Imports from package blenderproc are not grouped
29	31	convention	trailing-whitespace	C0303		Trailing whitespace
35	0	convention	trailing-whitespace	C0303		Trailing whitespace
41	0	convention	line-too-long	C0301		Line too long (147/120)
85	8	refactor	no-else-return	R1705	depth_at_points_via_raytracing	Unnecessary "else" after "return", remove the "else" and de-indent the code inside it
89	0	convention	trailing-whitespace	C0303		Trailing whitespace
90	0	convention	line-too-long	C0301		Line too long (134/120)
141	0	convention	trailing-whitespace	C0303		Trailing whitespace
159	7	convention	trailing-whitespace	C0303		Trailing whitespace
161	33	convention	trailing-whitespace	C0303		Trailing whitespace
165	0	convention	line-too-long	C0301		Line too long (132/120)
167	0	convention	trailing-newlines	C0305		Trailing newlines

Module blenderproc.python.loader.BlendLoader (blenderproc/python/loader/BlendLoader.py)

Line	Col.	Type	Symbol	ID	Obj	Message
27	0	convention	line-too-long	C0301		Line too long (130/120)
100	0	convention	trailing-whitespace	C0303		Trailing whitespace

Module blenderproc.python.loader.BopLoader (blenderproc/python/loader/BopLoader.py)

Line	Col.	Type	Symbol	ID	Obj	Message
353	0	convention	trailing-whitespace	C0303		Trailing whitespace

Module blenderproc.python.loader.CCMaterialLoader (blenderproc/python/loader/CCMaterialLoader.py)

Line	Col.	Type	Symbol	ID	Obj	Message
73	0	convention	line-too-long	C0301		Line too long (152/120)

Module blenderproc.python.postprocessing.PostProcessingUtility (blenderproc/python/postprocessing/PostProcessingUtility.py)

Line	Col.	Type	Symbol	ID	Obj	Message
15	0	convention	line-too-long	C0301		Line too long (137/120)

Module blenderproc.python.renderer.RendererUtility (blenderproc/python/renderer/RendererUtility.py)

Line	Col.	Type	Symbol	ID	Obj	Message
246	0	convention	trailing-whitespace	C0303		Trailing whitespace
314	0	convention	trailing-whitespace	C0303		Trailing whitespace
504	0	convention	trailing-whitespace	C0303		Trailing whitespace

Module blenderproc.python.types.EntityUtility (blenderproc/python/types/EntityUtility.py)

Line	Col.	Type	Symbol	ID	Obj	Message
170	11	convention	unidiomatic-typecheck	C0123	Entity.duplicate	Use isinstance() rather than type() for a typecheck.

Module blenderproc.python.types.LightUtility (blenderproc/python/types/LightUtility.py)

Line	Col.	Type	Symbol	ID	Obj	Message
125	12	refactor	consider-using-augmented-assign	R6104	Light.setup_as_projector	Use '/=' to do an augmented assign directly
216	0	convention	trailing-whitespace	C0303		Trailing whitespace

Module blenderproc.python.types.MeshObjectUtility (blenderproc/python/types/MeshObjectUtility.py)

Line	Col.	Type	Symbol	ID	Obj	Message
520	87	convention	trailing-whitespace	C0303		Trailing whitespace
526	0	convention	line-too-long	C0301		Line too long (122/120)
529	0	convention	trailing-whitespace	C0303		Trailing whitespace
535	68	warning	unused-variable	W0612	MeshObject.add_auto_smooth_modifier	Unused variable 'data_from'
543	16	warning	self-assigning-variable	W0127	MeshObject.add_auto_smooth_modifier	Assigning the same variable 'modifier' to itself
545	0	convention	trailing-whitespace	C0303		Trailing whitespace
555	0	warning	bad-indentation	W0311		Bad indentation. Found 9 spaces, expected 8
559	0	convention	trailing-whitespace	C0303		Trailing whitespace
561	0	warning	bad-indentation	W0311		Bad indentation. Found 9 spaces, expected 8
562	0	convention	trailing-whitespace	C0303		Trailing whitespace
564	0	warning	bad-indentation	W0311		Bad indentation. Found 9 spaces, expected 8
566	0	warning	bad-indentation	W0311		Bad indentation. Found 13 spaces, expected 12
567	0	warning	bad-indentation	W0311		Bad indentation. Found 13 spaces, expected 12
568	0	warning	bad-indentation	W0311		Bad indentation. Found 13 spaces, expected 12
569	90	convention	trailing-whitespace	C0303		Trailing whitespace
569	0	warning	bad-indentation	W0311		Bad indentation. Found 13 spaces, expected 12
570	0	warning	bad-indentation	W0311		Bad indentation. Found 13 spaces, expected 12
571	0	warning	bad-indentation	W0311		Bad indentation. Found 13 spaces, expected 12
572	0	convention	trailing-whitespace	C0303		Trailing whitespace
573	23	convention	trailing-whitespace	C0303		Trailing whitespace
574	0	warning	bad-indentation	W0311		Bad indentation. Found 9 spaces, expected 8
576	0	warning	bad-indentation	W0311		Bad indentation. Found 9 spaces, expected 8
577	20	convention	trailing-whitespace	C0303		Trailing whitespace
578	0	warning	bad-indentation	W0311		Bad indentation. Found 9 spaces, expected 8
579	0	convention	trailing-whitespace	C0303		Trailing whitespace
580	0	warning	bad-indentation	W0311		Bad indentation. Found 9 spaces, expected 8
612	0	convention	line-too-long	C0301		Line too long (130/120)
623	7	convention	trailing-whitespace	C0303		Trailing whitespace
631	8	warning	unused-variable	W0612	create_from_point_cloud	Unused variable 'p'
659	0	convention	trailing-whitespace	C0303		Trailing whitespace
734	28	convention	trailing-whitespace	C0303		Trailing whitespace

Module blenderproc.python.utility.Initializer (blenderproc/python/utility/Initializer.py)

Line	Col.	Type	Symbol	ID	Obj	Message
80	0	convention	trailing-whitespace	C0303		Trailing whitespace

Module blenderproc.python.utility.InstallUtility (blenderproc/python/utility/InstallUtility.py)

Line	Col.	Type	Symbol	ID	Obj	Message
129	0	convention	trailing-whitespace	C0303		Trailing whitespace

Module blenderproc.python.writer.BopWriterUtility (blenderproc/python/writer/BopWriterUtility.py)

Line	Col.	Type	Symbol	ID	Obj	Message
9	0	convention	wrong-import-order	C0411		standard import "from typing import List, Optional, Dict, Tuple" should be placed before "import trimesh"
10	0	convention	wrong-import-order	C0411		standard import "import warnings" should be placed before "import trimesh"
11	0	convention	wrong-import-order	C0411		standard import "import datetime" should be placed before "import trimesh"
18	0	convention	wrong-import-order	C0411		standard import "import sys" should be placed before "import trimesh"
62	0	convention	line-too-long	C0301		Line too long (131/120)
79	8	warning	redefined-outer-name	W0621	write_bop	Redefining name 'dataset_objects' from outer scope (line 536)
172	44	warning	protected-access	W0212	write_bop	Access to a protected member _pyrender_init of a client class
172	15	refactor	consider-using-with	R1732	write_bop	Consider using 'with' for resource-allocating operations
176	0	convention	trailing-whitespace	C0303		Trailing whitespace
182	0	convention	trailing-whitespace	C0303		Trailing whitespace
305	21	warning	redefined-outer-name	W0621	_BopWriterUtility.get_frame_gt	Redefining name 'dataset_objects' from outer scope (line 536)
392	38	warning	redefined-outer-name	W0621	_BopWriterUtility.write_frames	Redefining name 'dataset_objects' from outer scope (line 536)
518	0	convention	trailing-whitespace	C0303		Trailing whitespace
530	78	convention	trailing-whitespace	C0303		Trailing whitespace
534	8	warning	global-variable-undefined	W0601	_BopWriterUtility._pyrender_init	Global variable 'renderer' undefined at the module level
534	8	warning	global-variable-undefined	W0601	_BopWriterUtility._pyrender_init	Global variable 'renderer_large' undefined at the module level
534	8	warning	global-variable-undefined	W0601	_BopWriterUtility._pyrender_init	Global variable 'dataset_objects' undefined at the module level
552	0	convention	line-too-long	C0301		Line too long (173/120)
567	78	convention	trailing-whitespace	C0303		Trailing whitespace
573	8	warning	global-variable-not-assigned	W0602	_BopWriterUtility._calc_gt_masks_iteration	Using global for 'renderer' but no assignment is done
573	8	warning	global-variable-not-assigned	W0602	_BopWriterUtility._calc_gt_masks_iteration	Using global for 'dataset_objects' but no assignment is done
580	0	convention	trailing-whitespace	C0303		Trailing whitespace
670	0	convention	line-too-long	C0301		Line too long (160/120)
671	0	convention	trailing-whitespace	C0303		Trailing whitespace
672	0	convention	trailing-whitespace	C0303		Trailing whitespace
675	0	convention	line-too-long	C0301		Line too long (196/120)
688	11	convention	trailing-whitespace	C0303		Trailing whitespace
690	78	convention	trailing-whitespace	C0303		Trailing whitespace
696	8	warning	global-variable-not-assigned	W0602	_BopWriterUtility._calc_gt_info_iteration	Using global for 'renderer_large' but no assignment is done
696	8	warning	global-variable-not-assigned	W0602	_BopWriterUtility._calc_gt_info_iteration	Using global for 'dataset_objects' but no assignment is done
696	8	warning	global-variable-not-assigned	W0602	_BopWriterUtility._calc_gt_info_iteration	Using global for 'renderer' but no assignment is done
700	11	warning	protected-access	W0212	_BopWriterUtility._calc_gt_info_iteration	Access to a protected member _renderer of a client class
701	12	warning	protected-access	W0212	_BopWriterUtility._calc_gt_info_iteration	Access to a protected member _renderer of a client class
702	12	warning	protected-access	W0212	_BopWriterUtility._calc_gt_info_iteration	Access to a protected member _renderer of a client class
709	0	convention	trailing-whitespace	C0303		Trailing whitespace
837	0	convention	line-too-long	C0301		Line too long (202/120)
838	0	convention	trailing-whitespace	C0303		Trailing whitespace
846	44	warning	redefined-outer-name	W0621	_BopWriterUtility.calc_gt_coco	Redefining name 'dataset_objects' from outer scope (line 536)

Module blenderproc.python.writer.GifWriterUtility (blenderproc/python/writer/GifWriterUtility.py)

Line	Col.	Type	Symbol	ID	Obj	Message
13	0	convention	wrong-import-position	C0413		Import "from blenderproc.scripts.visHdf5Files import vis_data" should be placed at the top of the module
14	0	convention	wrong-import-position	C0413		Import "from blenderproc.python.utility.Utility import Utility" should be placed at the top of the module

Module blenderproc.scripts.quickstart (blenderproc/scripts/quickstart.py)

Line	Col.	Type	Symbol	ID	Obj	Message
1	0	convention	missing-module-docstring	C0114		Missing module docstring
11	-1	warning	pointless-string-statement	W0105		String statement has no effect
13	0	convention	wrong-import-position	C0413		Import "import numpy as np" should be placed at the top of the module

Metrics

Count per types

Name	Count
convention	64
refactor	3
warning	34

Count per messages

Name	Count
trailing-whitespace	40
line-too-long	13
trailing-newlines	1
no-else-return	1
ungrouped-imports	1
unused-import	1
consider-using-augmented-assign	1
unidiomatic-typecheck	1
bad-indentation	13
self-assigning-variable	1
unused-variable	2
redefined-outer-name	4
protected-access	4
consider-using-with	1
global-variable-undefined	3
global-variable-not-assigned	5
wrong-import-order	4
wrong-import-position	3
missing-module-docstring	1
pointless-string-statement	1

Count per modules

Name	Count
blenderproc	1
blenderproc.python.camera.CameraProjection	13
blenderproc.python.loader.CCMaterialLoader	1
blenderproc.python.loader.BopLoader	1
blenderproc.python.loader.BlendLoader	2
blenderproc.python.postprocessing.PostProcessingUtility	1
blenderproc.python.renderer.RendererUtility	3
blenderproc.python.types.LightUtility	2
blenderproc.python.types.EntityUtility	1
blenderproc.python.types.MeshObjectUtility	31
blenderproc.python.utility.InstallUtility	1
blenderproc.python.utility.Initializer	1
blenderproc.python.writer.BopWriterUtility	38
blenderproc.python.writer.GifWriterUtility	2
blenderproc.scripts.quickstart	3

Count per path

Name	Count
blenderproc/__init__.py	1
blenderproc/python/camera/CameraProjection.py	13
blenderproc/python/loader/CCMaterialLoader.py	1
blenderproc/python/loader/BopLoader.py	1
blenderproc/python/loader/BlendLoader.py	2
blenderproc/python/postprocessing/PostProcessingUtility.py	1
blenderproc/python/renderer/RendererUtility.py	3
blenderproc/python/types/LightUtility.py	2
blenderproc/python/types/EntityUtility.py	1
blenderproc/python/types/MeshObjectUtility.py	31
blenderproc/python/utility/InstallUtility.py	1
blenderproc/python/utility/Initializer.py	1
blenderproc/python/writer/BopWriterUtility.py	38
blenderproc/python/writer/GifWriterUtility.py	2
blenderproc/scripts/quickstart.py	3

***The tests are done.*** Check the log:

Last update: 09/27/2024, 17:50:59

[johan-apes]

@cornerfarmer

thank you, and sorry for delayed response. Your solution definitely is more elegant and cleaner. I have tested it and it works well. I just have two observations.

1. Collections that are not linked to the original scene will get linked to the scene
2. Visibility toggles - only hide_render and hide_viewport are preserved other toggles are ignored, which may result in different output between the scenes

[cornerfarmer]

Hey @johan-apes

thanks for checking the PR! Regarding your observations:

1. That is correct. Currently every collection that is not linked to any other collection will be linked to the scene after import. That is necessary, as it is not done automatically, even if the collection was originally linked to the scene in the imported file. So with that, we cannot really distinguish that case from the one where the scene was not linked to the scene in the original file. But is there even a use case of importing a collection and not wanting it to be linked to the scene?
2. Which other toggles do you mean? Were these toggles preserved with the code from your PR?

[johan-apes]

1. I agree, the behavior makes sense, but felt like I needed to at least let it be known
2. I meant these:

tbh it was not handled in my PR either, however I noticed it now during testing, we sometimes have use cases where we have hidden collection that we use kind of a junkyard for wip objects and other stuff you do not necessarily need but don't want to delete.

If they would pop up in the scene the resulting image would look unexpected.

[cornerfarmer]

Hi @johan-apes,

thanks for the explanation. If the other toggles are not preserved, I think there is not really something we can do, as this is then more of a blender bug. I would now merge this PR, but if you find a way to preserve all toggle, feel free to create a new PR.