godot-benchmarks icon indicating copy to clipboard operation
godot-benchmarks copied to clipboard
verified publisher icon godotengine

[TRACKER] Benchmarks to create

Open Calinou opened this issue 3 years ago • 11 comments

⚠️⚠️NOTE⚠️⚠️ This list is outdated, please refer to the following one instead:

https://github.com/godotengine/godot-benchmarks/issues/36

⚠️⚠️END OF NOTE⚠️⚠️

This is a list of benchmarks to add to the benchmark suite. See the contributing guide for instructions on adding new benchmarks.

I plan to tackle some of those benchmarks personally, so please comment on this issue if you plan on working on new benchmarks (to avoid working on the same benchmark at the same time).

Feel free to suggest new benchmark ideas as well :slightly_smiling_face: Remember that we'd like to keep the project fast to clone, so the repository size needs to remain low. This means we can't do things like adding large detailed 3D scenes with textures, as these usually require several GBs of storage.

2D Rendering

  • [x] 50 Sprite2Ds https://github.com/godotengine/godot-benchmarks/pull/18
  • [x] 500 Sprite2Ds https://github.com/godotengine/godot-benchmarks/pull/18
  • [x] 5,000 Sprite2Ds https://github.com/godotengine/godot-benchmarks/pull/18
  • [x] 10 Polygon2Ds (32 points circle) https://github.com/godotengine/godot-benchmarks/pull/19
  • [x] 100 Polygon2Ds (32 points circle) https://github.com/godotengine/godot-benchmarks/pull/19
  • [x] 1,000 Polygon2Ds (32 points circle) https://github.com/godotengine/godot-benchmarks/pull/19
  • [x] DirectionalLight2D without shadows https://github.com/godotengine/godot-benchmarks/pull/60
  • [x] DirectionalLight2D with shadows https://github.com/godotengine/godot-benchmarks/pull/60
  • [x] 50 PointLight2Ds without shadows https://github.com/godotengine/godot-benchmarks/pull/60
  • [x] 50 PointLight2Ds with shadows https://github.com/godotengine/godot-benchmarks/pull/60
  • [x] GPUParticles2D (few nodes with lots of particles) https://github.com/godotengine/godot-benchmarks/pull/62
  • [x] GPUParticles2D (many nodes with few particles) https://github.com/godotengine/godot-benchmarks/pull/62
  • [x] CPUParticles2D (few nodes with lots of particles) https://github.com/godotengine/godot-benchmarks/pull/62
  • [x] CPUParticles2D (many nodes with few particles) https://github.com/godotengine/godot-benchmarks/pull/62
  • [ ] Small/simple TileMap
  • [ ] Small/simple TileMap with lights
  • [ ] Large/complex TileMap
  • [ ] Large/complex TileMap with lights

3D Rendering

Some of those benchmarks should have "exclude on Forward Mobile" and "exclude on Compatibility" properties so that they are skipped on unsupported renderers.

  • [ ] 50 BoxMeshes
  • [ ] 500 BoxMeshes
  • [ ] 5,000 BoxMeshes
  • [ ] 10 SphereMeshes
  • [ ] 100 SphereMeshes
  • [ ] 1,000 SphereMeshes
  • [ ] 10 skinned character meshes
  • [ ] 100 skinned character meshes
  • [ ] 1,000 skinned character meshes
  • [ ] 10 ReflectionProbes (up close)
  • [ ] 100 ReflectionProbes (far away)
  • [ ] 10 Decals (up close)
  • [ ] 100 Decals (far away)
  • [ ] DirectionalLight3D without shadows
  • [ ] DirectionalLight3D with shadows
  • [ ] 50 OmniLight3Ds without shadows
  • [ ] 50 OmniLight3Ds with shadows (non-PCSS)
  • [ ] 50 OmniLight3Ds with shadows (PCSS)
  • [ ] 50 SpotLight3Ds without shadows
  • [ ] 50 SpotLight3Ds with shadows (non-PCSS)
  • [ ] 50 SpotLight3Ds with shadows (PCSS)
  • [ ] GPUParticles3D (few nodes with lots of particles)
  • [ ] GPUParticles3D (many nodes with few particles)
  • [ ] CPUParticles3D (few nodes with lots of particles)
  • [ ] CPUParticles3D (many nodes with few particles)
  • [ ] Volumetric fog (global)
  • [ ] Volumetric fog + 10 FogVolumes
  • [ ] Volumetric fog + 100 FogVolumes

To compare the relative performance impact of each setting, these tests should render the same scene but with different settings:

Antialiasing

  • [ ] No AA
  • [ ] FXAA
  • [ ] TAA
  • [ ] 2× MSAA
  • [ ] 4× MSAA
  • [ ] 8× MSAA
  • [ ] TAA + FXAA
  • [ ] TAA + 2× MSAA
  • [ ] TAA + 4× MSAA
  • [ ] TAA + 8× MSAA

Global illumination

  • [ ] No GI
  • [ ] SDFGI (default settings)
  • [ ] SDFGI (8 cascades + lower cell size)
  • [ ] SDFGI (half resolution)
  • [ ] SDFGI (half resolution, 8 cascades + lower cell size)
  • [ ] VoxelGI
  • [ ] VoxelGI (half resolution, same scene as SDFGI)
  • [ ] LightmapGI (indirect only, non-directional)
  • [ ] LightmapGI (indirect only, directional)
  • [ ] LightmapGI (fully baked, non-directional)
  • [ ] LightmapGI (fully baked, directional)

Post-processing

  • [ ] None
  • [ ] SSR
  • [ ] SSAO
  • [ ] SSAO (half resolution)
  • [ ] SSAO + SSR
  • [ ] SSIL
  • [ ] SSIL (half resolution)
  • [ ] SSIL + SSR
  • [ ] SSAO + SSIL
  • [ ] SSAO (half resolution) + SSIL
  • [ ] SSAO + SSIL (half resolution)
  • [ ] SSAO (half resolution) + SSIL (half resolution)
  • [ ] SSAO + SSIL + SSR
  • [ ] Glow
  • [ ] Auto exposure
  • [ ] Fog (non-volumetric)
  • [ ] Tonemap (linear)
  • [ ] Tonemap (ACES)
  • [ ] Adjustments (no texture)
  • [ ] Adjustments (1D LUT)
  • [ ] Adjustments (3D LUT)

Core

  • [ ] Create 100,000 nodes (flat)
  • [ ] Create 100,000 nodes (nested as deep as possible)
  • [ ] Create + Rename 100,000 nodes (flat)
  • [ ] Create + Rename 100,000 nodes (nested as deep as possible)
  • [ ] Create + Remove 100,000 nodes (flat)
  • [ ] Create + Remove 100,000 nodes (nested as deep as possible)
  • [ ] Parse 100,000 Expressions (simple)
  • [ ] Parse 100,000 Expressions (complex)
  • [ ] Parse 100,000 Expressions (complex, with function call)

GDScript

https://github.com/godotengine/godot/tree/master/modules/gdscript/tests can be helpful for inspiration.

  • [ ] Create 1,000,000 untyped variables
  • [ ] Create 1,000,000 typed variables (explicit type)
  • [ ] Create 1,000,000 typed variables (with type inference)
  • [ ] Create/modify/delete small dictionary 100,000 times
  • [ ] Create/modify/delete large dictionary 100 times
  • [ ] Create custom class and instantiate it 100,000 times
  • [x] Create Callable and call it 100,000 times https://github.com/godotengine/godot-benchmarks/pull/32
  • [ ] Use load() on resource 100,000 times
  • [ ] Create 100,000 Colors from HTML strings
  • [ ] Go through long if 100,000 times
  • [ ] Go through long match 100,000 times (with same logic as if)
  • [ ] Go through heavily nested if 100,000 times
  • [ ] Go through heavily nested match 100,000 times
  • [ ] Long for loop
  • [ ] Long while loop (with same logic as for)
  • [x] String formatting (%, no-op) https://github.com/godotengine/godot-benchmarks/pull/32
  • [x] String formatting (%, simple) https://github.com/godotengine/godot-benchmarks/pull/32
  • [x] String formatting (%, complex) https://github.com/godotengine/godot-benchmarks/pull/32
  • [x] String formatting (String.format(), no-op) https://github.com/godotengine/godot-benchmarks/pull/32
  • [x] String formatting (String.format(), simple) https://github.com/godotengine/godot-benchmarks/pull/32
  • [x] String formatting (String.format(), complex) https://github.com/godotengine/godot-benchmarks/pull/32
  • [ ] var2str() on all core types 10,000 times
  • [ ] str2var() on all core types 10,000 times
  • [ ] Find whether a small number is prime (with recursion)
  • [ ] Find whether a medium number is prime (with recursion)
  • [ ] Find whether a large number is prime (with recursion)

Navigation

  • [ ] AStar2D
  • [ ] AStar3D
  • [ ] 2D pathfinding
  • [ ] 3D pathfinding

Physics

All tests should be done in 2D and 3D when possible. To only represent physics performance, no visual representation should be made unless a debug flag is passed (e.g. --debug-benchmarks).

  • [ ] 1,000 StaticBodies (box)
  • [ ] 100 box CharacterBodies (falling on StaticBody box)
  • [ ] 1,000 box CharacterBodies (falling on StaticBody box)
  • [ ] 1,000 StaticBodies (complex concave shape)
  • [ ] 100 box CharacterBodies (falling on StaticBody complex concave shape)
  • [ ] 1,000 box CharacterBodies (falling on StaticBody complex concave shape)

Repeat with box, sphere, capsule, cylinder, simple convex (quickhull) and complex convex (VHACD):

  • [ ] 100 RigidDynamicBodies (falling on StaticBody box, placed far apart to avoid collisions with other rigid bodies)
  • [ ] 100 RigidDynamicBodies (falling on StaticBody box, placed close to each other)
  • [ ] 1,000 RigidDynamicBodies (falling on StaticBody box, placed far apart to avoid collisions with other rigid bodies)
  • [ ] 1,000 RigidDynamicBodies (falling on StaticBody box, placed close to each other)
  • [ ] 100 RigidDynamicBodies (falling on StaticBody complex concave shape, placed far apart to avoid collisions with other rigid bodies)
  • [ ] 100 RigidDynamicBodies (falling on StaticBody complex concave shape, placed close to each other)
  • [ ] 1,000 RigidDynamicBodies (falling on StaticBody complex concave shape, placed far apart to avoid collisions with other rigid bodies)
  • [ ] 1,000 RigidDynamicBodies (falling on StaticBody complex concave shape, placed close to each other)

GUI

  • [ ] Containers (with lots of nodes)
  • [ ] RichTextLabel (with lots of lines and no BBCode)
  • [ ] RichTextLabel (with lots of lines and basic BBCode)
  • [ ] RichTextLabel (with lots of lines and complex nested BBCode)
  • [x] Label (with lots of text) https://github.com/godotengine/godot-benchmarks/pull/32

Calinou avatar Aug 09 '22 10:08 Calinou

I think I might have a go at the Sprite2D and Polygon2D benchmarks, just a quick question, though. Should I instance the sprites/polygons through code, or should they already be in the scene when the benchmark is run?

Mach565 avatar Nov 09 '22 21:11 Mach565

Should I instance the sprites/polygons through code, or should they already be in the scene when the benchmark is run?

There will be a lot of sprites/polygons to add, so you should instance them through code like the 3D benchmarks are doing.

Calinou avatar Nov 09 '22 21:11 Calinou

Alright, thanks. I'll start work on it now.

Mach565 avatar Nov 09 '22 21:11 Mach565

Sprite2D benchmarks should be ready to review, #18

Mach565 avatar Nov 11 '22 04:11 Mach565

Polygon2D benchmarks are ready, #19

Mach565 avatar Dec 03 '22 06:12 Mach565

Could I give the DirectionalLight2D and the PointLight2D tasks a try? Additionally, to get around the 2D light limit, should I instantiate multiple background sprites as well as multiple sprites with occluders?

Otherwise, we could end up with lights not rendering. image

Alternatively, I could toggle lights on and off, but I'm not sure which method would be preferred.

tinybitofheaven avatar Mar 25 '24 16:03 tinybitofheaven

Could I give the DirectionalLight2D and the PointLight2D tasks a try?

Sure, feel free to open a pull request :slightly_smiling_face:

Additionally, to get around the 2D light limit, should I instantiate multiple background sprites as well as multiple sprites with occluders?

Yes, we need to make sure all lights show up.

Calinou avatar Mar 26 '24 00:03 Calinou

The PointLight2D and DirectionalLight2D benchmarks are ready for review! #60

tinybitofheaven avatar Mar 27 '24 16:03 tinybitofheaven

Could I try the GPUParticles2D and CPUParticles2D benchmarks? And are there any specific particle settings for those that I should be setting? Or a specific number of nodes/particles?

leannelu avatar Apr 14 '24 16:04 leannelu

Could I try the GPUParticles2D and CPUParticles2D benchmarks?

Sure, go ahead :slightly_smiling_face:

And are there any specific particle settings for those that I should be setting? Or a specific number of nodes/particles?

I'd try creating the particle setups mentioned (that is, one benchmark with lots of particle nodes that have few particles each, and one benchmark with few particle nodes that have lots of particles each). As for the mesh/material, we want to be testing simulation performance as opposed to rendering performance, so I recommend using a small billboarded QuadMesh (0.1 × 0.1 units) with a material that has its shading mode set to Unshaded.

Calinou avatar Apr 15 '24 17:04 Calinou

GPUParticles2D and CPUParticles2D benchmarks are ready to review! #62

leannelu avatar Apr 21 '24 06:04 leannelu