Suggestion: Default group render order to not overwrite existing group order

[gkjohnson]

I feel that the behavior of the Group.renderOrder attribute is not all that intuitive. I made a topic on the forum here to discuss if you're interested in reading but my confusion stems from the fact the current behavior is that a Groups render order can only apply to the immediate child meshes and not nested child meshes.

In this case:

Group with render order -1
    └ Mesh A with render order 2
Mesh B with render order 0

Mesh A will render before Mesh B.

Inserting a new group like so without changing the default render order means that Mesh B will render before Mesh A:

Group with order -1
    └ Group with default order (0)
        └ Mesh A with render order 2
Mesh B with render order 0

However by default I would expect the default behavior to not affect the draw order. This happens because the WebGLRenderer always overwrites the last group order with no way of indicating that a group should not affect the sort. I think ideally a group with a new order would only affect the render order within that subtree (as opposed to globally as it does now) but that would be a much larger change and I think this is a step in the right direction. I can update the docs and do more testing if this has promise.

TL;DR This PR defaults the renderOrder for Group to be null to indicate that it should not affect the child meshes draw order.

Edit Added encoding of group order as string to account for whole hierarchy of parents draw orders.

[WestLangley]

IMO, the proper behavior would be for Mesh A to render before Mesh B in both of your examples.

All the objects within a group should be rendered together.

That is what I would expect of a sorting group.

[gkjohnson]

IMO, the proper behavior would be for Mesh A to render before Mesh B in both of your examples.

Oops you're right -- I've updated the above example to correctly describe the behavior I'm talking about.

All the objects within a group should be rendered together.

I agree I think that makes the most sense -- do you have any thoughts on how to achieve that? I suppose every set of children could be sorted before being rendered but that seems heavy handed and would be a big change. Another option and a smaller change might be to encode the render order of all parents as an array or string in projectObject so that can be used when sorting the meshes. The latter option means that children of parents of equal renderOrder could be mixed when being sorted but imo that's okay.

[gkjohnson]

@WestLangley I've updated the PR to show how encoding the group hierarchy as a string might look so objects can be sorted taking the hierarchy of parents render order into account.

If any of this looks promising I can look into benchmarking performance.

Any thoughts? I'd really like to help solve this because at the moment it's really painful to work around controlling the draw order of geometry.

[WestLangley]

@gkjohnson Thanks for working on this!

I stand by what I said above. I expect there can be edge cases, but I'd expect it to work correctly in 'common' use cases.

I will defer implementation detail suggestions to someone better suited than myself. @Mugen87 has good ideas on these things.

[gkjohnson]

Great!

I stand by what I said above. I expect there can be edge cases, but I'd expect it to work correctly in 'common' use cases.

Okay if, as you're suggesting, every subgroup should be sorted separately then I'd like to clarify a couple things:

• If Groups will sort all immediate children then should Object3Ds do the same? If not then all nested children of an Object3D (until a Group is found) will be sorted together in contrast to how a Group behaves. In this case it feels like a pretty big and unexpected difference in the way that the two objects behave.

• How should transparent object sorting work? It may be important to specify the group and render order for transparent objects but by default all transparent objects should be rendered from back to front so you would want them to be all sorted together by default, which would be more similar to the behavior I described.

[WestLangley]

Hmm... This is a bit complicated... I think you should get buy-in from @mrdoob before you invest a lot of time in this. He will have to decide what behavior he wants.

Users can always emulate sorting groups by having multiple scenes.

[gkjohnson]

Users can always emulate sorting groups by having multiple scenes.

Unfortunately this isn't really a suitable workaround, either, because lighting information from the primary scene can't be used when rendering a subsequent one. In order to do this properly you'd have to copy all the lights to the new scene with objects you want to render second and keep them in sync every frame.

I'm hoping there's some middle ground we can find that simplifies this because it's something I'm struggling with at the moment.

[gkjohnson]

@mugen87 Sorry for taking a bit to describe my use case. I'm building an application for visualizing robotics and their environment along with an API that allows users to add extra objects to the scene hierarchy. One of these additions involves highlighting the terrain or drawing shapes on top of it that do not affect the robot model. To achieve this I would like to guarantee that the terrain draw first, then the highlights, then the robot.

To simplify the API I've created helper objects that set up stencil materials and an object subhierarchy for rendering which could be multiple groups deep. Likewise the terrain geometry can also be composed of multiple meshes that are grouped several deep which means that Groups resetting the render order becomes problematic. Ideally our users would be able to compose these objects without having to worry about managing and setting every objects render order in the scene and instead just rely on the root "Terrain" groups render order when adding it.

At the core what I would like to be able to do is create classes that can guarantee a draw order of a set of objects for the sake of making it easy to reuse. I've put together a fiddle to illustrate what I'm talking about:

https://jsfiddle.net/rcvef3wL/2/

Note that the cube (representing the robot) in the middle of the scene is not supposed to be highlighted red. You can uncomment some lines at the end of the init function. Hopefully the fiddle is clear enough. If not please let me know I'll try to elaborate more clearly. Keep in mind this is a simplified use case. As these highlights and other models are drawn it's important to be able to control the order of them all relative to each other.

@WestLangley I see now how not sorting every groups children as a set can be problematic because the sibling objects that are intended to be drawn together will be intermixed unless the parent group is given a unique renderOrder to its siblings. Maybe an additional object like RenderGroup or something similar could be introduced to accommodate this use case where children need to be sorted and drawn as a set? Or transparent objects could just not take the group render order into account and instead a custom blend material could be used for order-dependent alpha blended materials.

Thank you!

[mrdoob]

For reference, this behaviour was introduced in #15484.

[gkjohnson]

After thinking about this a bit more I think it might be best to develop a new class for sub-hierarchy-rendering rather than trying to reinvent the current one so here are a few thoughts on what one could look like.

I like the idea of a RenderGroup object that renders all sub objects together. Here's an example of how it might behave:

const meshA = new Mesh( /* ... */ );
meshA.renderOrder = 0;

const meshB = new Mesh( /* ... */ );
meshB.renderOrder = 1;

const renderGroup = new RenderGroup();
renderGroup.renderOrder = 2;

const meshC = new Mesh( /* ... */ );
meshC.renderOrder = 3;

const group = new Group();

renderGroup.add( meshA );
scene.add( renderGroup );

group.add( meshB );
scene.add( group );

scene.add( meshC );

renderer.render( scene, camera );

// scene
//   - renderGroup
//     - meshA
//   - group
//     - meshB
//   - meshC

// RenderGroup.renderOrder is compared against Mesh.renderOrder
// Renders meshB, meshA, meshC

This has the benefits of not affecting the render algorithm if a RenderGroup is not used in the scene, control of the exact render order of a set of meshes / render groups, and which objects are rendered together. This means that objects can be created that encapsulate and dictate the order in which their children render without having to consider rest of the scene.

Regarding materials marked as transparent = true -- I think they should remain unaffected by the RenderGroup.renderOrder and continue to be sorted by depth. Or if it's important that transparent objects be rendered together maybe the RenderGroup can include an option such as excludeTransparent that would exclude transparent objects from being affected by the group.

I'm happy to give implementing this a go in another PR but it would be great to get feedback on the desired behavior first!

@WestLangley @Mugen87 @mrdoob

[Mugen87]

Can you please explain the difference between RenderGroup and SortingGroup (which was discussed here #14433)?

The current logic of Group.renderOrder was enhanced in order to avoid the introduction of a new class (see https://github.com/mrdoob/three.js/pull/14433#issuecomment-450405281).

[gkjohnson]

Can you please explain the difference between RenderGroup and SortingGroup (which was discussed here #14433)

I did see that PR -- the behavior wasn't immediately clear to me looking at it because it wasn't described but after taking a closer look it does seem to achieve what I'm describing aside from the transparency flag. I would also like to try to figure out how to get rid of the flag on WebGLRenderer to enable the behavior.

The current logic of Group.renderOrder was enhanced in order to avoid the introduction of a new class (see #14433 (comment)).

I guess I'm in part asking if this is still a requirement. While I think the above described behavior can be achieved without creating a new class I feel it may be more clear to introduce a second class. Either one would work, though.

If the above suggestion and behavior in #14433 is agreeable (with or without a new class) then I may have some ideas on how to implement it without the need for the sortingGroupsEnabled flag.

[gkjohnson]

I've created #18599 which implements what I've described using a different approach from #14433 -- feedback appreciated!

[mrdoob]

Hmm... What if we changed Object3D.renderOrder to null by default? 🤔

[gkjohnson]

Hmm... What if we changed Object3D.renderOrder to null by default? 🤔

This is another option but it's not enough to just modify the current behavior. The behavior in #18599 should be used if renderOrder is not null. But otherwise there's no difference. Setting the renderOrder of a group to a number would effectively mean "change this into a RenderGroup".

I thought making RenderGroup separate would make the difference in behavior more apparent and deliberate but that's just a matter of personal preference. I can update the other PR to make this change.

[gkjohnson]

@mrdoob

tl;dr I recommend a flag like Group.sortChildren = true to enable to RenderGroup behavior instead because overloading renderOrder to achieve it is confusing.

---

I've now modified #18599 in a different branch to use Group.renderOrder instead and I actually find it very confusing and uncomfortable to use compared to an separate "RenderGroup" because setting the renderOrder to a number changes the behavior too significantly.

I'll use the sprite case originally brought up in #14415 to explain. In that case you want to make sure the two sprites render next right after each other to avoid incorrect overlap, so considering the following case:

const circleSprite = new Mesh( circleGeometry, circleMaterial );
const ringSprite = new Mesh( ringGeometry, ringMaterial );

const sprite1 = new Group();
sprite1.add( circleSprite.clone(), ringSprite.clone() );

const sprite2 = new Group();
sprite2.add( circleSprite.clone(), ringSprite.clone() );

scene.add( sprite1, sprite2 );

In the above case you'll wind up with the overlap issues demonstrated in the original example. However setting the render order of each sprite to 0 will trigger the "render all children together" behavior and address the overlap:

const circleSprite = new Mesh( circleGeometry, circleMaterial );
const ringSprite = new Mesh( ringGeometry, ringMaterial );

const sprite1 = new Group();
sprite1.add( circleSprite.clone(), ringSprite.clone() );
sprite1.renderOrder = 0;

const sprite2 = new Group();
sprite2.renderOrder = 0;
sprite2.add( circleSprite.clone(), ringSprite.clone() );

scene.add( sprite1, sprite2 );

It feels odd that setting the renderOrder to all the same value would fix the issue because they were all the same in the first place. Maybe instead of defaulting renderOrder to null and changing it to trigger the behavior we can add another flag to Group? Something like Group.renderChildrenTogether or Group.sortChildren would be more explicit and clear without creating a new class. If Group.renderChildrenTogether is set to true then renderOrder would have an effect otherwise renderOrder is not used on Group.

[mrdoob]

Sorry, I'm a bit overwhelmed. Lets go step by step.

Hmm... What if we changed Object3D.renderOrder to null by default? 🤔

If we did this, then your second example would turn into:

Group with order -1
    └ Group with default order (null)
        └ Mesh A with render order 2
Mesh B with render order 0

In the renderer we can then ignore Group.renderOrder in that case by changing this:

https://github.com/mrdoob/three.js/blob/e0e541ba1ff246a84c3e947991dc54fad21dbe0d/src/renderers/WebGLRenderer.js#L1299-L1302

Into this:

if ( object.isGroup && object.renderOrder !== null ) {

	groupOrder = object.renderOrder;

[gkjohnson]

Sorry, I'm a bit overwhelmed. Lets go step by step.

Heh sorry, I misunderstood your suggestion 😇

I see now you're suggesting making a smaller change to the existing behavior to allow the group order to propagate down to all the children. I tried that approach previously, as well, and it's still confusing and insufficient in the cases I'm looking at.

To use the simpler example from #14415, imagine you have two sprites that you need to render right after one another to avoid sorting issues. I would like to create a class that guarantees sort order regardless of what else is in the scene and what other object renderOrders are. However with the given Group.renderOrder suggestion there's no way to do that without setting the render order of the parent group and making sure you don't set it to a value that's already being used or between values that are already being used for an existing effect. I feel that this should just work:

class ExampleSprite extends Group {

    constructor(...args) {

        super(...args);
        const circleSprite = new Mesh( circleGeometry, circleMaterial );
        const ringSprite = new Mesh( ringGeometry, ringMaterial );

        this.add( circleSprite, ringSprite );
    
    }

}

for ( let i = 0; i < 1000; i ++ ) {

    const sprite = new ExampleSprite();
    sprite.position.set( Math.random(), Math.random(), 0 );
    scene.add( sprite );

}

But with Group.renderOrder as it is (even defaulting to null) it will not. It requires a delicate dance of ensuring that all renderOrder values in the scene are harmonious which means it's also difficult or impossible to create an encapsulated object with an effect that uses nested Group.renderOrders. It's also easy to break if a renderOrder is changed to a wrong value or set improperly. The fact that a new Group.renderOrder is used to sort globally rather than relative to the parent group makes this more complicated. Overall I think continuing to sort Group.renderOrder globally and not giving the option to sort and render children together is a halfway solution to a more robust and understandable one that I think is achievable.

I know this inherently winds up touching sensitive parts of the codebase so I'm happy to try to explain or chunk up the code a bit more after we make sure the behavior at a high level is agreeable :D

[gkjohnson]

@mrdoob

Any further thoughts on this? Hopefully I've explained the behavior well enough. I'm happy to work up to #18599 more incrementally like you mentioned but I want to make sure the desired behavior and API is agreed upon.