rework renderer api around command buffers and render passes

[i509VCB]

Right now the Renderer trait relies on the Binding a target and then calling Renderer::render. This works well with the GL based renderer API where the current target is managed by the state machine. However this is suboptimal for Vulkan where the goal is to record several renderpasses (which are effectively describing a target being bound) into a single command buffer and submitting the whole command buffer.

The current GL-like API means that for 5 targets, Vulkan submits 5 command buffers to the queue. Command buffer recording is one of the most expensive tasks for a Vulkan driver, so making larger buffers is more performant. Plus this only needs one submission, which means you do not need to wait on 5 fences.

The GL-like bind and render api can be implemented in a Vulkan-like command buffer and render pass form by deferring execution of commands until Frame::finish.

There is a good reason to make this the default API style for Renderer: Using a vulkan specific API would require specialization for any existing code that is generic (RenderElement for example) in order to have the more efficient codepath.

This style however raises the question whether a software renderer API would like this API or if it would conflict. This would make sense for a software renderer since each pass could borrow a buffer to render to. But that would either require being able to clone to target or ensure the lifetime of the buffer is longer than the command buffer (and until execution finishes).

The API would look something like this:

pub trait Renderer {
    // ... associated types

    // instead of calling frame several times, a command buffer is recorded.
    fn record_command_buffer(&mut self) -> Result<Self::CommandBuffer<'_, Self>, Error>;
}

pub trait CommandBuffer<'a, R: Renderer> {
    // Starting a render pass also binds the specified target for the scope of the render pass.
    // Even when dropped, the commands are still recorded. 
    fn start_pass<T>(&mut self, target: &T) -> Result<R::Renderpass<'_>, Error>
    where
        R: Bind<T>;

    // submit could return a sync object in the future to allow waiting.
    //
    // Like Frame::finish, you need to submit a command buffer for anything to happen.
    fn submit(&mut self) -> Result<(), Error>;
}

pub trait RenderPass<'a> {}

[Drakulix]

So this basically sounds like you want to move the Bind api into the Frame?

Also seems vaguely related to https://github.com/Smithay/smithay/issues/806.

[phire]

It might be unwise to directly expose the CommandBuffer concept. It might be better to expose a more generic abstraction.

I've been scheming, and it should be possible to make a a vulkan backend that reuses existing command buffers as much as possible. There are many possible implementation, but I think in the absolute best case (with a single compute shader and VK_KHR_buffer_device_address), it should be possible to reuse the same static command buffer almost indefinitely.