Add 3D Viewport Containers and Nodes

[nashiora]

I'd like to actually try to work on this as it's been my biggest reason for not working with o!f for my rhythm game projects.

For most of the rulesets I'd like to work on in o!f that require 3D space a very simple implementation could suffice: unlit primitives or simple meshes with minimally programmable materials, such as changing colors for portions of textures based on user settings, with different blend modes (alpha and additive being the primary two.) I'd imagine at a bare minimum the following would be starting points:

• A new Container as the root node for the scene, defining the viewport as its bounds on the screen. Like any other container, it simply contains its children and is positioned on the screen. That bounding area is the total space of the framebuffer render target.

• A Camera, for which I feel like the simplest approach is that a camera is actually built into the new Container and not a separate instance at all, meaning the viewport and defined world-space properties exist together; a separate camera instance would imply the eventual ability to render to a separate framebuffer to me, which is a much larger scope than the container itself being the only possible render target.

• A Drawable and DrawNode base type. Ideally, this could inherit the existing Drawables and DrawNodes, though a significant amount of their implementation exists solely in the 2D world and would be useless or confusing to extend or work around.

• It's much easier to define the graphics of 2D shapes as simple properties, but depending on the scope of 3D implementation a Material system may be needed. The easiest solution being only supporting primitive 3D shapes and parameterizing their rendering similar to the 2D drawables, the more future proof, of course, being Meshesand Materials.

I'm not familiar with o!f's code beyond what I studied of it a few years back, so I don't know the scope of these features or where to start for most of them, or if a better system is possible and floating in someone's mind. Another consideration is whether or not lighting should be considered, or at least programmable in a material by the user with enough world and vertex information, as some shapes or play areas may need the depth and contours.

[smoogipoo]

I'm glad someone's willing to attempt this. I've been thinking about how this would all work for a while now, so this is a good opportunity to get everything in writing. I'm by no means an expert on 3D graphics data structures, but hey the 2D system of o!f didn't start without issue either.

Firstly, I'm going to be using the word "Model" to depict a 3D object. I know this is a bit confusing, but I have no better name for it other than "Drawable3D" and that's even worse imo.

In general, your idea is very similar to the direction I was hoping to take, with a few changes. This is a huge task so I'll try to document exactly the changes required as greatly as possible.

Specific requirements:

• All 3D rendering must occur inside a frame-buffer. The back-buffer is reserved for 2D depth optimisations.
• The hierarchical structure of o!f should be preserved.

Structure

Structure is the most important part of this to me. A good first step I think is to split out CompositeDrawable into one or more interfaces with default methods exposing common functionality. For example:

interface IDrawable
    LoadState LoadState { get; }

interface ICompositeDrawable : IDrawable
    IReadOnlyList<Drawable> InternalChildren { get;}

interface ILifetimeManagingComposite : ICompositeDrawable
    bool UpdateChildrenLife()
    {
        ...
    }

From there, I'd like to see the following structure take place:

ICompositeDrawable
    -> ICompositeDrawable<TDrawable> : IComposite

Drawable
    -> CompositeDrawable : Drawable, ICompositeDrawable<Drawable>
        -> Container<Drawable> : CompositeDrawable
    -> Model : Drawable
        -> CompositeModel : Model, ICompositeDrawable<Model>
            -> Container<Model> : CompositeModel
    -> Scene : Drawable, ICompositeDrawable<Model>

The Model class

This is the basis of the 3D hierarchy. It takes many of the properties of Drawable and hides them, exposing 3D variants in return. For example:

class Model : Drawable
{
    Vector3 position;

    new Vector3 Position
    {
        get => position;
        set
        {
            position = value;
            base.Position = value.Xy;
            Depth = value.Z;
        }
    }

    float Depth { get; set;}
}

Eventually we can separate this out in a similar way to ICompositeDrawable, however it's not super important for the time being.

The CompositeModel class

This is a composite that only accepts Models to be added to it. It contains all the children/lifetime management/autosize/masking capabilities you'd expect minus a few that are specific to the 2D hierarchy such as padding/borders/corners.

The purpose of this class is to facilitate 3D operations of a group of Models.

The Scene class

This is a composite similar to CompositeModel however it is not a Model itself. This distinction is important because this acts as the root of the 3D hierarchy and cannot normally be nested within the 3D hierarchy.

The purpose of this class is to set up the pipeline appropriately for 3D drawing, including creating the afore-mentioned framebuffer and the perspective projection.

BufferedContainerDrawNode can probably be inherited from here, with some extra setup done before children are drawn to the framebuffer.

The ModelWrapper class

class ModelWrapper : Model
    ctor(Drawable child);

I haven't mentioned this above, but this is something we'll probably want. It's a Model which receives a single Drawable in its constructor.

The purpose of this class is to draw the child to a framebuffer, and then render that framebuffer in a 3D space.

Example

class MyScreen : Screen
    public MyScreen()
    {
        InternalChildren = new Drawable[]
        {
            new Box // Background
            {
                RelativeSizeAxes = Axes.Both,
                Colour = Color4.Green
            },
            new Scene
            {
                Anchor = Anchor.Centre,
                Origin = Anchor.Centre,
                RelativeSizeAxes = Axes.Both,
                Size = new Vector2(0.5f),
                Children = new Drawable[]
                {
                    new ModelWrapper(new FillFlowContainer // A flat circle background
                    {
                        RelativeSizeAxes = Axes.Both,
                        ChildrenEnumerable = Enumerable.Range(1, 100).Select(_ => new Circle { Size = new Vector2(20) })
                    })
                    {
                        Position = new Vector3(0, 0, 1f)
                    },
                    new Cube
                    {
                        Anchor = Anchor.Centre,
                        Origin = Anchor.Centre,
                        Size = new Vector3(200, 200, 0.2f),
                        Position = new Vector3(0, 0, 0.5f)
                    },
                }
            }
        }
    }

Materials and lighting

As you said, we'll definitely want these.

Lighting would be implemented as a derived ModelContainer that provides a custom DrawNode + properties + shader implementing the lighting model.

Materials I see being done as an interface or an abstract implementation of Model itself. Since the material shader is dependent on a parenting lighting container, if such a parent does not exist then the models could be parameterised via the current pipeline shaders.

[nashiora]

I'm going to read thru the code and get myself up to speed on how Drawables already work. What parts of Drawables (or CompositeDrawables) will actually be useful for Model / Scene? A lot of it looks very 2D specific and I'm curious how it'll be changed or overwritten, especially if the only place a Model is valid is in a CompositeModel / Scene, where only the newed members matter.

Otherwise everything you've mentioned sounds solid, I'll have more to ask as I learn the code better before doing anything.

[smoogipoo]

From CompositeDrawable:

• Lifetime management.
• Async loading.
• The tasks involved in adding/removing/clearing children.
• Recursive transform functions (ApplyTransformsAt, etc).
• Recursive DrawNode generation (addFromComposite).
• Invalidation.
• Input queue construction and related members.
• Masking (just the Masking bool). The implementation of UpdateSubTreeMasking can remain in the 2D hierarchy (it'll probably need to be reimplemented in 3D).
• Relative child sizing/offsetting. These can probably also be omitted and remain in the 2D hierarchy, but will eventually need to be reimplemented in 3D.
• Auto-sizing. This can remain part of the 2D hierarchy since it'll need to be reimplemented for 3D.
• The entirety of UpdateSubTree.

[Flutterish]

I am actually working on a VR port of osu!lazer at the moment and I have already implemented (basic) shaders, raycast physics, panels that display 2D drawables and a separate 3D draw node system. I think it might be useful for this goal, so I will publish my progress as soon as I get to a nice state with it. This is how it currently looks like:

[nashiora]

@Flutterish Has this progress been published?

[Flutterish]

o!xr and the 2 derived projects o!f-xr and openvr.net are planned to go open source within 2 next releases.

[ALiwoto]

o!xr and the 2 derived projects o!f-xr and openvr.net are planned to go open source within 2 next releases.

nice

[nashiora]

Either way I'm looking to tackle this again, finally. I've been itching to do 3D rhythm game work again and the idea of building my own framework from scratch again when I'm just going to lack the wonderful features o!f already has, esp since I want to work in C#, is tiring to continue with.

@smoogipoo Has anything substantial changed with how you envision this? Or with the framework itself that would change what you previously designed in this comment?

[smoogipoo]

@nashiora My vision hasn't changed, but it's idealistic and you're free to work on/propose something different.