Script and tutorial for generating procedural datasets with Blender

[AndrejOrsula]

🎉 New feature

Summary

Hello :smile:. I recently saw https://github.com/ignitionrobotics/ign-gazebo/pull/1401 so I thought that I would share how I have used Blender to generate procedural datasets for one of my projects. The video below should explain the gist of it.

https://user-images.githubusercontent.com/22929099/160182025-e1868b54-f3f1-44ee-9ea6-b62e181e8af5.mp4

For this request, I modified my existing setup into a script that directly exports SDF models (inspired by Ignition tutorial for exporting SDF exporting from Blender). I included a "tutorial" to go alongside it with hope that other contributors would share their .blend files in the future. This tutorial contains the important details, so I won't repeat them here.

I am not exactly sure what would be the best way to integrate this "feature" with Ignition Fuel without uploading terabytes of datasets. Maybe it could be discussed below if this request is of any interest.

Test it

• Install Blender 3.0 or 3.1 (or try your luck with other versions)
• Open rock.blend
• Execute the embedded Python script
• See the results

Note: rock.blend was my first attempt at utilising Geometry Nodes. The procedural pipeline is therefore far from optimal. Maybe using one of the official demos from https://www.blender.org/download/demo-files/#geometry-nodes and modifying it for dataset generation would be better to serve as an example.

Checklist

• [x] Signed all commits for DCO
• [ ] Added tests
• [x] Added example and/or tutorial
• [x] Updated documentation (as needed)
• [ ] Updated migration guide (as needed)
• [ ] codecheck passed (See contributing)
• [ ] All tests passed (See test coverage)
• [ ] While waiting for a review on your PR, please help review another open pull request to support the maintainers

Note to maintainers: Remember to use Squash-Merge and edit the commit message to match the pull request summary while retaining Signed-off-by messages.

[codecov[bot]]

Codecov Report

Merging #1412 (9ba5b22) into ign-gazebo6 (3106687) will increase coverage by 0.50%. The diff coverage is 72.01%.

:exclamation: Current head 9ba5b22 differs from pull request most recent head cc210d1. Consider uploading reports for the commit cc210d1 to get more accurate results

@@               Coverage Diff               @@
##           ign-gazebo6    #1412      +/-   ##
===============================================
+ Coverage        62.95%   63.46%   +0.50%     
===============================================
  Files              301      307       +6     
  Lines            24256    24783     +527     
===============================================
+ Hits             15271    15729     +458     
- Misses            8985     9054      +69     

Impacted Files	Coverage Δ
include/ignition/gazebo/EntityComponentManager.hh	100.00% <ø> (ø)
include/ignition/gazebo/ServerConfig.hh	100.00% <ø> (ø)
include/ignition/gazebo/Util.hh	100.00% <ø> (ø)
include/ignition/gazebo/detail/BaseView.hh	100.00% <ø> (ø)
src/BaseView.cc	100.00% <ø> (ø)
.../plugins/component_inspector/ComponentInspector.hh	28.57% <ø> (ø)
src/gui/plugins/component_inspector/Pose3d.hh	0.00% <0.00%> (ø)
src/rendering/SceneManager.cc	28.27% <0.00%> (+2.21%)	:arrow_up:
...int_position_controller/JointPositionController.hh	100.00% <ø> (ø)
...rc/systems/odometry_publisher/OdometryPublisher.hh	100.00% <ø> (ø)
... and 32 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 3106687...cc210d1. Read the comment docs.

[AndrejOrsula]

It seems to be "relatively" simple to adapt an existing Geometry Nodes setup from projects made by talented artists such as Tree Generator by Blenderesse. Full-featured material conversion (with texture baking) are still missing in the script to get a proper transfer though (hence the colourless leaves).

---

I also tried to transfer the wind animation (shown below from Blender). However, Ignition Gazebo actors utilise skeleton-based animations as far as I can tell. With Geometry Nodes, this is not the case (at least for the animation below). The displacement of geometry is based directly on the animation frame number that drives the wind effect... so there is no armature or anything like that. Therefore, exporting of animations made in Geometry Nodes is probably a no-go, or at least not so straight-forward.

https://user-images.githubusercontent.com/22929099/160861465-55639d4c-da47-4567-9f15-21e6e9acd690.mp4

[mabelzhang]

Beautiful! We can possibly mock up some corals for underwater with that, I imagine. Could you suggest to the artist to do a generator for schools of fish or jellyfish? :D

Thank you for the contribution and recommendation!

(Sorry haven't had a chance to do the review yet.)

[AndrejOrsula]

No worries about the review. Some of the important features are still missing (TODOs are included inside the script). I also cannot say that all edge-cases are covered (definitely not tested), so there might be several ways in which a Blender project isn't supported. However, I decided to open this request now anyway because I wouldn't look into it for at least several months otherwise.

Certain missing features should be relatively easy to implement (e.g. rendering of thumbnails), but a proper support for exporting materials with procedural textures will require a lot more work and a possible refactoring. I suppose the end-goal would be to have an SDF-exporter addon in Blender that would support also the generation of datasets. It sounds like a much larger project though...

---

Could you suggest to the artist to do a generator for schools of fish or jellyfish? :D

Here is a great tutorial for schools of fish by Ian Hubert: Animate Fish in Blender - Lazy Tutorials [1:00] (Big Ol Swooshes!)

[mayman99]

Thanks for the PR @AndrejOrsula and great idea.

Did you come across BlenderProc before? it is a procedural Blender pipeline for data generation. it has are a position sampler and physics positioning modules already, meaning if we just have a world-sdf writer, we could create a dataset using it or use one of the excising (including a dataset for rock), and just add our world sdf exporter on top.

[AndrejOrsula]

Hello @mayman99,

Thanks for your reply.

Did you come across BlenderProc before? it is a procedural Blender pipeline for data generation.

Yes, BlenderProc sounds great! However, I have not tried it yet. Therefore, please correct me if I am wrong; As far as I could tell, it was/is targeting data generation for visual (un)supervised learning by providing multi-channel rendered images (with some labels such as semantics). My use-case is reinforcement learning where robots need to interact with the environment, so a dataset of images would not suffice. For this reason, I did not find it applicable for what I tried to achieve.

It has a position sampler and physics positioning modules already, meaning if we just have a world-sdf writer, we could create a dataset using it or use one of the excising (including a dataset for rock), and just add our world sdf exporter on top.

I did not know about the physics positioning module. That's neat! Regarding the world-sdf writer, I think sdf_exporter.py tries to achieve a similar thing - but currently supports only static worlds.

Integration of BlenderProc with Gazebo/SDF sounds like a great idea! However, I believe that having the entire world inside a single SDF description can be quite limiting for dynamic scenes, especially if the aim is to randomize the environment (domain randomization). So exporting individual models might provide more flexibility. If the end-use is simulation in Gazebo, I also think that sampling random position of robots/objects directly inside Gazebo is more computationally efficient (rather than deleting the world and loading a new one that is generated by BlenderProc).

I am not familiar with the feature set of BlenderProc when it comes to randomizing geometry and/or material of objects. I found Blender's Geometry Nodes very nice to work with and I believe it has a great potential for robot learning inside simulations, hence this PR. But if there is already something similar in BlenderProc as well, that would be great!

[mayman99]

The tutorial lgtm, thanks! but the rock.blend has a cube not a rock, so the output I got after running the script was just normal cubes, perhaps its the wrong file? or is the script supposed to turn the cube into a rock shape?

[mayman99]

Yes, BlenderProc sounds great! However, I have not tried it yet. Therefore, please correct me if I am wrong; As far as I could tell, it was/is targeting data generation for visual (un)supervised learning by providing multi-channel rendered images (with some labels such as semantics). My use-case is reinforcement learning where robots need to interact with the environment, so a dataset of images would not suffice.

Atm BlenderProc does what you described, but what I'm proposing is adding a "writer" to BlenderProc so it would write the scenes as SDF as well, I've opened an issue there for that. that explain my point.

I did not know about the physics positioning module. That's neat! Regarding the world-sdf writer, I think sdf_exporter.py tries to achieve a similar thing - but currently supports only static worlds.

This is a good point, I still haven't had the chance to get a deeper look into the script, but it sounds it wouldn't be very trivial.

I am not familiar with the feature set of BlenderProc when it comes to randomizing geometry and/or material of objects. I found Blender's Geometry Nodes very nice to work with and I believe it has a great potential for robot learning inside simulations, hence this PR. But if there is already something similar in BlenderProc as well, that would be great!

Randomizing materials and textures in blenderProc is there, not so sure about geometry nodes.

I could see us integrating BlenderProc with gazebo in the future, but as for now I would need to learn more about SDF worlds to understand what is possible and what is not.

[AndrejOrsula]

... rock.blend has a cube not a rock, so the output I got after running the script was just normal cubes, perhaps its the wrong file?

I just checked the file and it is the correct one. What Blender version are you running? I tested only Blender 3.0/3.1, and I do not believe any older version would be functional.

Can you see Geometry Nodes under object's modifiers? Cube is the original geometry that is subdivided and processed through this modifier (no other processing is done to the object - everything is done by the nodes).

or is the script supposed to turn the cube into a rock shape?

The script is unrelated to the geometry generation process.

---

what I'm proposing is adding a "writer" to BlenderProc so it would write the scenes as SDF as well, I've opened an https://github.com/DLR-RM/BlenderProc/issues/539 that explain my point.

Great initiative!

I wonder how a generic SDF exporter class for Blender (briefly discussed above in one of the review comments) could fit into such writer for BlenderProc. I don't know whether BlenderProc uses some internal states that would need to be supported, or states available through the Blender Python API would suffice for exporting (while supporting all features of BlenderProc). For example, if a generic SDF exporter is feature-rich and supports exporting of full scenes as an SDF world - then maybe it could be called from within BlenderProc after the scene is generated/randomized/distorted?

[codecov[bot]]

Codecov Report

Merging #1412 (5a4f1f1) into ign-gazebo6 (e502e9f) will increase coverage by 0.03%. The diff coverage is n/a.

:exclamation: Current head 5a4f1f1 differs from pull request most recent head 26686cf. Consider uploading reports for the commit 26686cf to get more accurate results

@@               Coverage Diff               @@
##           ign-gazebo6    #1412      +/-   ##
===============================================
+ Coverage        64.65%   64.69%   +0.03%     
===============================================
  Files              321      321              
  Lines            26275    26055     -220     
===============================================
- Hits             16988    16855     -133     
+ Misses            9287     9200      -87     

Impacted Files	Coverage Δ
src/SystemLoader.cc	63.51% <0.00%> (-3.96%)	:arrow_down:
src/SystemManager.cc	97.08% <0.00%> (-0.56%)	:arrow_down:
src/systems/hydrodynamics/Hydrodynamics.cc	88.07% <0.00%> (-0.46%)	:arrow_down:
src/Util.cc	93.11% <0.00%> (-0.33%)	:arrow_down:
src/Conversions.cc	84.05% <0.00%> (-0.13%)	:arrow_down:
src/rendering/RenderUtil.cc	38.90% <0.00%> (-0.12%)	:arrow_down:
include/ignition/gazebo/SystemLoader.hh	100.00% <0.00%> (ø)
.../systems/triggered_publisher/TriggeredPublisher.hh	100.00% <0.00%> (ø)
...s/multicopter_motor_model/MulticopterMotorModel.cc	75.93% <0.00%> (+0.21%)	:arrow_up:
.../plugins/component_inspector/ComponentInspector.cc	5.61% <0.00%> (+0.30%)	:arrow_up:
... and 3 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[mabelzhang]

@AndrejOrsula Friendly ping! The code looks good, just need some more explanation in the tutorial so that readers have more context about what the code is doing, and the Ignition -> Gazebo rename. Let us know if you have time to address it.

We'd happily do the renaming for you, but for the tutorial, I don't have enough understanding of the code to write the details.

[AndrejOrsula]

Thank you so much @mabelzhang for your review!

Apologies, I have not had time to look at it yet.

I will make sure to address your comments and finalize this PR by the end of this week. I will ping you once I am done. Thank you in advance.

[AndrejOrsula]

Thank you again, @mabelzhang, for the review!

I went through your comments and tried to address all of them. Here are some notable changes since last time.

1. Updated the tutorial with better instructions/examples and made the explanation more detailed. Please check. :smiley:

• Link

2. Improved the script and its default parameters. Also fixed some issues and used the proper Wavefront (.obj) exporter such that most materials get exported correctly. Made this exported default for visual geometry because the COLLADA exporter works terribly on certain models for some reason (material issues - I am not sure why).

• Almost all default parameters are now at the top of the script so that these can be easily configured on a per-model basis in each .blend project.

3. Automatic generation of thumbnails (for Fuel).

• Only a single render of the 3D viewport, but I think it is good enough for now. It is disabled when the script is run with Blender in the background (-b).

4. Included a garden.blend file demo for generating procedural terrains with scattered low-poly trees, rocks, grass and flowers. Also reduced the size of the rock.blend demo project by compressing it and removing unnecessary embedded files.

• The exported models of garden.blend do not really look like a typical garden, but it is the closest I could manage in one afternoon with my limited knowledge while learning how Geometry Nodes work. It looks quite cartoonish because of the low-resolution meshes and no textures, but it allows Gazebo to run reasonably fast even if terrain with a large size is generated. Separate collision geometry is also automatically exported (not for grass and flower for obvious reason -- only for ground, trees and large rocks).
• I reused a few assets from Blender Studio that are all licensed under CC0, so I hope that is fine (e.g. tree generator and meshes for flowers). I modified them to support randomization with Seed input attribute.
• Let me know if the naming of the file (garden.blend) is not good due to conflicts with the upcoming release
• Future idea: Maybe we could also create a procedural generator of gazebo SDF models for Gazebo to go alongside these garden SDF models in Gazebo Garden! :laughing:

https://user-images.githubusercontent.com/22929099/187162834-57a48d0a-cb35-446a-98c8-a05185823753.mp4

• The demo with rocks is still there

• Please, let me know if I should upload some of these models to Fuel (or whether I should refrain from it due to their large size, as people can generate the models themselves). I also do not know if there is a place to host such files in the future in case more people would be interested in them. The garden.blend size is only 328 KB, and rock.blend is 1.37 MB, so I don't think it is an issue to have these demos under the repository.

5. Added CLI argument parsing to the script.

• I did not include the full list of CLI arguments in the tutorial. This is the full list. Please, let me know if I should include it in there under some section. (The tutorial already contains the command to show this message)

usage: 
        blender [blender options] file.blend --python-text script.py -- [script options]
        blender [blender options] file.blend --python external_script.py -- [script options]
list script options:
        blender -b file.blend --python-text script.py -- -h
        blender -b file.blend --python external_script.py -- -h
list blender options:
        blender -h
example:
        blender rock.blend --python-text procedural_dataset_generator.py -- -o sdf_models/rock

Generate a procedural dataset of SDF models using Blender.

options:
  -h, --help            show this help message and exit
  --export-dataset      [default mode] Generate a procedural dataset. (default: True)
  --export-model        [alternative mode] Export a single model.
  -o OUTPUT_DIR, --output-dir OUTPUT_DIR
                        The output directory for all models (each model is located under its own subdirectory). When exporting a single SDF model, its output path is 'OUTPUT_DIR/MODEL_NAME'. For datasets, the path of each SDF model name is appended by the random seed used during its generation as 'OUTPUT_DIR/MODEL_NAME+VARIANT_SEED'. If set to empty, Fuel cache '${GZ_FUEL_CACHE_PATH}/fuel.gazebosim.org/${USER}/models' is used. (default: /home/andrej/sdf_models)
  -v MODEL_VERSION, --model-version MODEL_VERSION
                        The version of exported model path that is joined with the model output path 'OUTPUT_DIR/MODEL_NAME/MODEL_VERSION', or 'OUTPUT_DIR/MODEL_NAME+VARIANT_SEED/MODEL_VERSION'. The output path is modified only if 'MODEL_VERSION' is set (not None). For negative values and an existing model directory, the next unique version is used to avoid overwriting. If 'OUTPUT_DIR' is empty (export to Fuel cache), a negative value is used to guarantee a unique
                        version. (default: None)
  --model-name-prefix MODEL_NAME_PREFIX
                        Prefix of exported model as 'OUTPUT_DIR/MODEL_NAME_PREFIX+MODEL_NAME' , or 'OUTPUT_DIR/MODEL_NAME_PREFIX+MODEL_NAME+VARIANT_SEED' (default: '').
  --model-name-suffix MODEL_NAME_SUFFIX
                        Suffix of exported model as 'OUTPUT_DIR/MODEL_NAME+MODEL_NAME_SUFFIX' , or 'OUTPUT_DIR/MODEL_NAME+MODEL_NAME_SUFFIX+VARIANT_SEED' (default: '').
  -s FIRST_SEED, --first-seed FIRST_SEED
                        The random seed of the first model when generating a dataset. (default: 0)
  -n NUMBER_OF_VARIANTS, --number-of-variants NUMBER_OF_VARIANTS
                        Number of model variants to export when generating a dataset. (default: 8)
  --filetype-visual {collada,dae,wavefront,obj,stl}
                        The format of exported visual geometry ['collada' == 'dae', 'wavefront' == 'obj']. (default: wavefront)
  --filetype-collision {collada,dae,wavefront,obj,stl}
                        The format of exported collision geometry ['collada' == 'dae', 'wavefront' == 'obj']. (default: stl)
  --detail-level-visual DETAIL_LEVEL_VISUAL
                        Level of detail for exported visual geometry, e.g. subdivision level. (default: 1)
  --detail-level-collision DETAIL_LEVEL_COLLISION
                        Level of detail for exported collision geometry, e.g. subdivision level. (default: 0)
  --ignore-objects-visual IGNORE_OBJECTS_VISUAL [IGNORE_OBJECTS_VISUAL ...]
                        List of objects to ignore when exporting visual geometry. (default: [])
  --ignore-objects-collision IGNORE_OBJECTS_COLLISION [IGNORE_OBJECTS_COLLISION ...]
                        List of objects to ignore when exporting collision geometry. (default: [])
  --symlink-external-textures SYMLINK_EXTERNAL_TEXTURES
                        If true, symbolic links will be created for all textures instead of copies. No copy or symlink will be if the texture is already under the output path. (default: True)
  --texture-source-mode {none,path,online,blender}
                        The source from which to select/extract (PBR) textures for the model. Option 'none' disables texturing in SDF and relies on mesh exporters. Option 'path' should either point to a single set of textures or a number of texture sets, from which a set would be sampled at random. Options 'online' (textures from an online source) and 'blender' (baking of textures) are currently not implemented. The value must be specified using the
                        'TEXTURE_SOURCE_VALUE' option. (default: none)
  --texture-source-value TEXTURE_SOURCE_VALUE
                        Value for the texture source, with the context based on the selected 'TEXTURE_SOURCE_MODE'. For example, this value expresses path to a directory with textures or a name of the environment. (default: None)
  --material-texture-diffuse MATERIAL_TEXTURE_DIFFUSE [MATERIAL_TEXTURE_DIFFUSE ...]
                        Diffuse intensity of the albedo texture map. Please, enter values for each channel as `--material-texture-diffuse R G B`. (default: (1.0, 1.0, 1.0))
  --material-texture-specular MATERIAL_TEXTURE_SPECULAR [MATERIAL_TEXTURE_SPECULAR ...]
                        Specular intensity of the albedo texture map. Please, enter values for each channel as `--material-texture-specular R G B`. (default: (0.2, 0.2, 0.2))
  --static STATIC       If true, the SDF model is exported as immovable and it won't be updated by the physics engine. (default: False)
  --inertial-estimation-mode {none,density,random_density,mass,random_mass}
                        The mode used during the estimation of inertial properties. Option 'none' disables estimation of inertial properties. Option '[random_]density' assumes a uniform density of the model. Option '[random_]mass' determines a uniform density based on the target mass. Random options uniformly sample the target value from a specified range. The value must be specified using the 'INERTIAL_ESTIMATION_VALUE' option. Estimation of inertial properties is
                        always disabled for 'STATIC' models. (default: none)
  --inertial-estimation-value INERTIAL_ESTIMATION_VALUE [INERTIAL_ESTIMATION_VALUE ...]
                        Value for the inertial estimation, with the context based on the selected 'INERTIAL_ESTIMATION_MODE'. For non-random modes, please use a single value as `--inertial-estimation-value TARGER_VALUE`. For a random range, please enter min and max values as `--inertial-estimation-value MIN MAX`. (default: None)
  --inertial-estimation-use-collision-mesh INERTIAL_ESTIMATION_USE_COLLISION_MESH
                        If true, collision geometry will be used for the estimation of inertial properties instead of the visual geometry of the model. (default: True)
  --friction-coefficient FRICTION_COEFFICIENT [FRICTION_COEFFICIENT ...]
                        Coefficient of the surface friction, equal in both directions. For a random range, please enter min and max values as `--friction-coefficient MIN MAX`. (default: [1.0])
  --generate-thumbnails GENERATE_THUMBNAILS
                        If true, thumbnails will be generated for the exported models. Only applicable if Blender is run in the foreground without `-b`. (default: True)

[mabelzhang]

Thank you for iterating! This is awesome! I'll read through the updated tutorial soon.

COLLADA exporter works terribly on certain models for some reason (material issues

Yes! I've seen that too. OBJ worked better for me as well.

Automatic generation of thumbnails (for Fuel).

Thank you!

Let me know if the naming of the file (garden.blend) is not good due to conflicts with the upcoming release

Good point! woodland.blend maybe? I think we have very small risk of getting to Gazebo W :sweat_smile:

procedural generator of gazebo SDF models for Gazebo to go alongside these garden SDF models in Gazebo Garden

Heh... <.<

I'll take a look at these when I do an actual review:

Updated the tutorial

licensed under CC0

[AndrejOrsula]

Good point! woodland.blend maybe? I think we have very small risk of getting to Gazebo W :sweat_smile:

Thank you for the suggestion! Name woodland.blend is great, I did not think about it. I renamed it and all its references (https://github.com/gazebosim/gz-sim/pull/1412/commits/5a4f1f185e0f1e754d1534db1f7393ad751f7485).

I also added a short disclaimer that the scattered assets used within this Blender file were adapted from Blender Studio (https://github.com/gazebosim/gz-sim/pull/1412/commits/1de663d719d2dfd5a5e84846d03cb695e4701888), just to make it more transparent.

[mabelzhang]

DCO is failing. It'll have to pass before we can merge.

[AndrejOrsula]

Thank you for the review!

DCO seems to fail due to 1 commit from applying review suggestions directly via GitHub UI (https://github.com/gazebosim/gz-sim/pull/1412/commits/7fdb05f577dc249b8ac05f6a015650f4480249c4). I am not exactly sure how to fix that in a non-destructive way.

Might be this, but I would like to confirm with you first.

git rebase -i -x "git commit --amend --no-edit --signoff" HEAD~10

If not, would you have some other suggestions? Thank you.

[mabelzhang]

Ah the web UI button for directly committing suggestions is tricky to use when there's DCO. For future reference, when you want to use that button again, you can manually copy your sign-off line from another commit, into the commit message in the web UI.

I can't say for sure. The best way to try it is to make a test branch locally, based on this branch, and try it on the test branch. If the result isn't what you want, you haven't messed up anything here.

If that command doesn't work for you, you can squash all your commits and force push. We will squash-merge the PR anyway, so that is not too bad. Since this is a new tutorial, if squashing using git gets too annoying, you can manually diff your branch against the target branch, and just do a single fresh commit, and force push.

Sorry about that detail. It's messed me up before too.

[AndrejOrsula]

Thank you for your suggestions and advice! I think this line did the trick. I removed the exec lines from all other commits that were already signed-off (probably not necessary).

git rebase -i -x "git commit --amend --no-edit --signoff" HEAD~10

[AndrejOrsula]

Presentation slides from ROSCon 2022 Lightning Talk: ROSCon22_gz-sim_1412.pdf (without embedded videos)

[mjcarroll]

CI errors here should all be false positives as this doesn't actually touch code.