chaoticbob
D3D12 colour RT transitions
This is one is more tricky than #15, but still in the same vein. While making an app that renders to a render target, and then binds that RT's colour attachment as a resource in a compute shader, this generates a D3D12 resource state error:
D3D12 ERROR: CGraphicsCommandList::SetComputeRootDescriptorTable: Resource state (0x4: D3D12_RESOURCE_STATE_RENDER_TARGET) (assumed at first use) of resource (0x00000217ED65DCD0:'Main pass RT') (subresource: 0) is invalid for use as a NON_PIXEL_SHADER_RESOURCE. Expected State Bits (all): 0x40: D3D12_RESOURCE_STATE_NON_PIXEL_SHADER_RESOURCE, Assumed Actual State: 0x4: D3D12_RESOURCE_STATE_RENDER_TARGET, Missing State: 0x40: D3D12_RESOURCE_STATE_NON_PIXEL_SHADER_RESOURCE. This validation is being enforced at bind during command list recording because the descriptor for this resource in the root signature is declared as DATA_STATIC_WHILE_SET_AT_EXECUTE. [ EXECUTION ERROR #538: INVALID_SUBRESOURCE_STATE]
I was surprised, because I added the correct tr_cmd_render_target_transition() calls, but upon investigation, the implementation of tr_internal_dx_cmd_render_target_transition() seems to be hard-coded to attachment #0 and present/RT states, and somehow only deals with the single-sample attachments? I'm not even sure how multisampling is supposed to work here.
May I get some explanation as to why the code is the way it is? The commit attached to this PR does its job and gets the passes working in my application, but it's bugging me how it's going to interact with other code using tinyrenderers.
This seems to be poor carry over from a copy/paste that originally handled multiple sampled swapchains. It works because of the genericity of the tr_render_target but as you pointed out wouldn't really work for RTs with more than one attachment. The same bug exists in the Vulkan implementation - but that code doesn't seem to be use in favor of Vulkna's auto resolve.
It seems like to fix this properly both tr_internal_dx_cmd_render_target_transition and tr_internal_dx_cmd_end_render need to account for multiple attachments. Additionally tr_cmd_render_target_transition should be extended to support transition of selected attachments.
May I ask, what are the cases you're using the render targets for and whether a more generic tr_cmd_render_target_transition that can support individual or ALL attachments would make sense?
In retrospect render_target might have been a poorly chosen name.
I created an Issue https://github.com/chaoticbob/tinyrenderers/issues/20 to clarify handling of render target attachment transitions and multi sample attachments.