kompute
kompute copied to clipboard
Published
20 hours ago •
KomputeProject
KomputeProject
Validation Error ：VUID-vkBeginCommandBuffer-commandBuffer-00050
Calling record() after calling eval() on the same kp:: sequence object will cause a vuid-vkbegincommandbuffer-commandbuffer-00050 Vulkan validation error.
Error occur in README.md demo:
...
// 4. Run operation synchronously using sequence
mgr.sequence()
->record<kp::OpTensorSyncDevice>(params)
->record<kp::OpAlgoDispatch>(algorithm) // Binds default push consts
->eval() // Evaluates the two recorded operations
->record<kp::OpAlgoDispatch>(algorithm, pushConstsB) // Overrides push consts ,<--- vuid-vkbegincommandbuffer-commandbuffer-00050
->eval(); // Evaluates only last recorded operation
...
Kompute create vulkan command pool without VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT, but kp::Sequence::begin will call vkBeginCommandBuffer(if command buffer is not in the initial state, the validation layer will get a error message ),
Will such error cause problems? Or should record() not be used after kp:: sequence:: eval()?
@cracy3m thank you very much for reporting this. At first glance this does seem to work as intended, you can see the test TestPushConstants.TestConstantsWrongSize which basically ensures that constants should not have different size to the one initially set up with.
If indeed this is not different size then it may be a bug, is this basically running the example in the demo? What is the version of your vulkan driver and headers? If it is indeed a bug we'll be keen to fix it.
For the time being you could use the workaround as show in the tests under TestPushConstants as you will see that there are examples that we validate for working push constants that are run directly with eval:
TEST(TestPushConstants, TestConstantsAlgoDispatchOverride)
{
{
std::string shader(R"(
#version 450
layout(push_constant) uniform PushConstants {
float x;
float y;
float z;
} pcs;
layout (local_size_x = 1) in;
layout(set = 0, binding = 0) buffer a { float pa[]; };
void main() {
pa[0] += pcs.x;
pa[1] += pcs.y;
pa[2] += pcs.z;
})");
std::vector<uint32_t> spirv = compileSource(shader);
std::shared_ptr<kp::Sequence> sq = nullptr;
{
kp::Manager mgr;
std::shared_ptr<kp::TensorT<float>> tensor =
mgr.tensor({ 0, 0, 0 });
std::shared_ptr<kp::Algorithm> algo = mgr.algorithm(
{ tensor }, spirv, kp::Workgroup({ 1 }), {}, { 0.0, 0.0, 0.0 });
sq = mgr.sequence()->eval<kp::OpTensorSyncDevice>({ tensor });
// We need to run this in sequence to avoid race condition
// We can't use atomicAdd as swiftshader doesn't support it for
// float
sq->eval<kp::OpAlgoDispatch>(algo,
std::vector<float>{ 0.1, 0.2, 0.3 });
sq->eval<kp::OpAlgoDispatch>(algo,
std::vector<float>{ 0.3, 0.2, 0.1 });
sq->eval<kp::OpTensorSyncLocal>({ tensor });
EXPECT_EQ(tensor->vector(), std::vector<float>({ 0.4, 0.4, 0.4 }));
}
}
}
My test context: platform: windows 10 (64bit) gpu: rtx 2070 vulkan sdk: 1.2.198.1 vulkan driver: I do not how to find the info, kompute version: v0.8.1
My test code:
void kompute(const std::string& shader) {
// 1. Create Kompute Manager with default settings (device 0, first queue and no extensions)
kp::Manager mgr;
// 2. Create and initialise Kompute Tensors through manager
// Default tensor constructor simplifies creation of float values
auto tensorInA = mgr.tensor({ 2., 2., 2. });
auto tensorInB = mgr.tensor({ 1., 2., 3. });
// Explicit type constructor supports uint32, int32, double, float and bool
auto tensorOutA = mgr.tensorT<uint32_t>({ 0, 0, 0 });
auto tensorOutB = mgr.tensorT<uint32_t>({ 0, 0, 0 });
std::vector<std::shared_ptr<kp::Tensor>> params = {tensorInA, tensorInB, tensorOutA, tensorOutB};
// 3. Create algorithm based on shader (supports buffers & push/spec constants)
kp::Workgroup workgroup{3, 1, 1}; // c++11 std::array 不能用括号加{}初始化
std::vector<float> specConsts({ 2 });
std::vector<float> pushConstsA({ 2.0 });
std::vector<float> pushConstsB({ 3.0 });
std::vector<uint32_t> orgspirv = compileSource(shader);
// std::vector<uint32_t> tspriv;
// tspriv.insert(tspriv.begin(),
// reinterpret_cast<const uint32_t *>(spirv_demo),
// reinterpret_cast<const uint32_t *>(spirv_demo + sizeof (spirv_demo)) + 1
// );
std::vector<uint32_t> tspriv(spirv_demo,
spirv_demo + sizeof (spirv_demo) / sizeof (spirv_demo[0]) //sizeof (decltype(spirv_demo[0]))
);
// std::vector<uint32_t> tspriv(ExtendAndWin,
// ExtendAndWin + sizeof (ExtendAndWin) / sizeof (ExtendAndWin[0]) //sizeof (decltype(spirv_demo[0]))
// );
KP_LOG_INFO("is compiled equel to showed : {}", orgspirv == tspriv);
auto algorithm = mgr.algorithm(params,
// See documentation shader section for compileSource
tspriv, //compileSource(shader),
workgroup,
specConsts,
pushConstsA);
// 4. Run operation synchronously using sequence
mgr.sequence()
->record<kp::OpTensorSyncDevice>(params)
->record<kp::OpAlgoDispatch>(algorithm) // Binds default push consts
->eval() // Evaluates the two recorded operations
->record<kp::OpAlgoDispatch>(algorithm, pushConstsB) // Overrides push consts
->eval(); // Evaluates only last recorded operation
// 5. Sync results from the GPU asynchronously
auto sq = mgr.sequence();
sq->evalAsync<kp::OpTensorSyncLocal>(params);
// ... Do other work asynchronously whilst GPU finishes
sq->evalAwait();
// fmt::print("{0:*^{1}}\n","result",30);
KP_LOG_INFO("{0:*^{1}}","result",30);
KP_LOG_INFO(fmt::format("tensorOutA: {}", fmt::join(tensorOutA->vector(), ", ")));
KP_LOG_INFO(fmt::format("tensorOutB: {}", fmt::join(tensorOutB->vector(), ", ")));
// // Prints the first output which is: { 4, 8, 12 }
// for (const float& elem : tensorOutA->vector()) std::cout << elem << " ";
// std::cout << std::endl;
// // Prints the second output which is: { 10, 10, 10 }
// for (const float& elem : tensorOutB->vector()) std::cout << elem << " ";
// std::cout << std::endl;
} // Manages / releases all CPU and GPU memory resources
int main()
{
// kpextSetLogger("kpextLogger", 1, 2);
// kpextSetLogger(nullptr, 1, 2);
#ifdef DEBUG
spdlog::set_level(spdlog::level::debug);
SPDLOG_LOGGER_DEBUG(spdlog::default_logger_raw(), "spdlog::default_logger_raw");
spdlog::set_level(spdlog::level::info);
KP_LOG_DEBUG("this debug log will not show \n");
KP_LOG_INFO("this info log will show","\n");
#endif
// spdlog::set_level(spdlog::level::off);
spdlog::set_level(spdlog::level::debug);
// Define a raw string shader (or use the Kompute tools to compile to SPIRV / C++ header
// files). This shader shows some of the main components including constants, buffers, etc
std::string shader = (R"(
#version 450
layout (local_size_x = 1) in;
// The input tensors bind index is relative to index in parameter passed
layout(set = 0, binding = 0) buffer buf_in_a { float in_a[]; };
layout(set = 0, binding = 1) buffer buf_in_b { float in_b[]; };
layout(set = 0, binding = 2) buffer buf_out_a { uint out_a[]; };
layout(set = 0, binding = 3) buffer buf_out_b { uint out_b[]; };
// Kompute supports push constants updated on dispatch
layout(push_constant) uniform PushConstants {
float val;
} push_const;
// Kompute also supports spec constants on initalization
layout(constant_id = 0) const float const_one = 0;
void main() {
uint index = gl_GlobalInvocationID.x;
out_a[index] += uint( in_a[index] * in_b[index] );
out_b[index] += uint( const_one * push_const.val );
}
)");
kompute(shader);
return 0;
}
Log info:
[2022-08-05 15:33:27.901] [debug] [Algorithm.cpp:325] Kompute Algorithm binding pipeline
[2022-08-05 15:33:27.901] [debug] [Algorithm.cpp:330] Kompute Algorithm binding descriptor sets
[2022-08-05 15:33:27.902] [debug] [Algorithm.cpp:346] Kompute Algorithm binding push constants memory size: 4
[2022-08-05 15:33:27.902] [debug] [Algorithm.cpp:360] Kompute Algorithm recording dispatch
[2022-08-05 15:33:27.904] [debug] [Sequence.cpp:95] Kompute sequence EVAL BEGIN
[2022-08-05 15:33:27.905] [debug] [Sequence.cpp:66] Kompute Sequence calling END
[2022-08-05 15:33:27.908] [info] [Sequence.cpp:77] Kompute Sequence command recording END
[2022-08-05 15:33:27.909] [debug] [OpTensorSyncDevice.cpp:42] Kompute OpTensorSyncDevice preEval called
[2022-08-05 15:33:27.909] [debug] [OpAlgoDispatch.cpp:49] Kompute OpAlgoDispatch preEval called
[2022-08-05 15:33:27.910] [debug] [Sequence.cpp:132] Kompute sequence submitting command buffer into compute queue
[2022-08-05 15:33:27.911] [debug] [OpTensorSyncDevice.cpp:48] Kompute OpTensorSyncDevice postEval called
[2022-08-05 15:33:27.912] [debug] [OpAlgoDispatch.cpp:55] Kompute OpAlgoDispatch postSubmit called
[2022-08-05 15:33:27.914] [debug] [kompute/operations/OpAlgoDispatch.hpp:32] Kompute OpAlgoDispatch constructor
[2022-08-05 15:33:27.915] [debug] [Sequence.cpp:279] Kompute Sequence record function started
[2022-08-05 15:33:27.915] [debug] [Sequence.cpp:39] Kompute sequence called BEGIN
[2022-08-05 15:33:27.915] [info] [Sequence.cpp:51] Kompute Sequence command now started recording
[2022-08-05 15:33:27.916] [debug] [Manager.cpp:26] [VALIDATION]: Validation - Validation Error: [ VUID-vkBeginCommandBuffer-commandBuffer-00050 ] Object 0: handle = 0x20044d1da40, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x27d60e0000000019, type = VK_OBJECT_TYPE_COMMAND_POOL; | MessageID = 0xb24f00f5 | Call to vkBeginCommandBuffer() on VkCommandBuffer 0x20044d1da40[] attempts to implicitly reset cmdBuffer created from VkCommandPool 0x27d60e0000000019[] that does NOT have the VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT bit set. The Vulkan spec states: If commandBuffer was allocated from a VkCommandPool which did not have the VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT flag set, commandBuffer must be in the initial state (https://vulkan.lunarg.com/doc/view/1.2.198.1/windows/1.2-extensions/vkspec.html#VUID-vkBeginCommandBuffer-commandBuffer-00050)
[2022-08-05 15:33:27.917] [debug] [Sequence.cpp:284] Kompute Sequence running record on OpBase derived class instance
[2022-08-05 15:33:27.917] [debug] [OpAlgoDispatch.cpp:21] Kompute OpAlgoDispatch record called
[2022-08-05 15:33:27.917] [debug] [Tensor.cpp:227] Kompute Tensor recording PRIMARY buffer memory barrier
I test TestPushConstants.TestConstantsWrongSize code, same error.
void test(){
{
std::string shader(R"(
#version 450
layout(push_constant) uniform PushConstants {
float x;
float y;
float z;
} pcs;
layout (local_size_x = 1) in;
layout(set = 0, binding = 0) buffer a { float pa[]; };
void main() {
pa[0] += pcs.x;
pa[1] += pcs.y;
pa[2] += pcs.z;
})");
std::vector<uint32_t> spirv = compileSource(shader);
std::shared_ptr<kp::Sequence> sq = nullptr;
{
kp::Manager mgr;
std::shared_ptr<kp::TensorT<float>> tensor =
mgr.tensor({ 0, 0, 0 });
std::shared_ptr<kp::Algorithm> algo = mgr.algorithm(
{ tensor }, spirv, kp::Workgroup({ 1 }), {}, { 0.0, 0.0, 0.0 });
sq = mgr.sequence()->eval<kp::OpTensorSyncDevice>({ tensor });
// We need to run this in sequence to avoid race condition
// We can't use atomicAdd as swiftshader doesn't support it for
// float
sq->eval<kp::OpAlgoDispatch>(algo,
std::vector<float>{ 0.1f, 0.2f, 0.3f }); //VUID-vkBeginCommandBuffer-commandBuffer-00050
sq->eval<kp::OpAlgoDispatch>(algo,
std::vector<float>{ 0.3f, 0.2f, 0.1f }); //VUID-vkBeginCommandBuffer-commandBuffer-00050
sq->eval<kp::OpTensorSyncLocal>({ tensor }); //VUID-vkBeginCommandBuffer-commandBuffer-00050
// EXPECT_EQ(tensor->vector(), std::vector<float>({ 0.4, 0.4, 0.4 }));
}
}
}
int main()
{
// kpextSetLogger("kpextLogger", 1, 2);
// kpextSetLogger(nullptr, 1, 2);
#ifdef DEBUG
spdlog::set_level(spdlog::level::debug);
SPDLOG_LOGGER_DEBUG(spdlog::default_logger_raw(), "spdlog::default_logger_raw");
spdlog::set_level(spdlog::level::info);
KP_LOG_DEBUG("this debug log will not show \n");
KP_LOG_INFO("this info log will show","\n");
#endif
// spdlog::set_level(spdlog::level::off);
spdlog::set_level(spdlog::level::debug);
test();
...
Invoke stack:
1 kp::debugMessageCallback Manager.cpp 27 0x7ff698978c12
2 debug_log_msg vk_layer_logging.h 481 0x7fff46e01731
3 LogMsgLocked vk_layer_logging.h 735 0x7fff46da94aa
4 ValidationObject::LogError chassis.h 3394 0x7fff46da843d
5 CoreChecks::PreCallValidateBeginCommandBuffer core_validation.cpp 6984 0x7fff46dad207
6 vulkan_layer_chassis::BeginCommandBuffer chassis.cpp 2659 0x7fff46c20c96
7 vk::DispatchLoaderStatic::vkBeginCommandBuffer vulkan.hpp 2089 0x7ff6989c9532
8 vk::CommandBuffer::begin<vk::DispatchLoaderStatic> vulkan_funcs.hpp 4467 0x7ff6989c45f8
9 kp::Sequence::begin Sequence.cpp 53 0x7ff6989c1093
10 kp::Sequence::record Sequence.cpp 283 0x7ff6989c0238
11 kp::Sequence::eval Sequence.cpp 104 0x7ff6989c0529
12 kp::Sequence::eval<kp::OpAlgoDispatch,std::vector<float>> Sequence.hpp 139 0x7ff6989e5ad7
13 test main.cpp 163 0x7ff6989fd199
14 main main.cpp 289 0x7ff6989fe3b4
Log :
...
[2022-08-05 15:53:21.887] [debug] [OpTensorSyncDevice.cpp:42] Kompute OpTensorSyncDevice preEval called
[2022-08-05 15:53:21.887] [debug] [Sequence.cpp:132] Kompute sequence submitting command buffer into compute queue
[2022-08-05 15:53:21.890] [debug] [OpTensorSyncDevice.cpp:48] Kompute OpTensorSyncDevice postEval called
[2022-08-05 15:53:24.688] [debug] [kompute/operations/OpAlgoDispatch.hpp:32] Kompute OpAlgoDispatch constructor
[2022-08-05 15:53:24.688] [debug] [Sequence.cpp:86] Kompute Sequence calling clear
[2022-08-05 15:53:24.688] [debug] [Sequence.cpp:279] Kompute Sequence record function started
[2022-08-05 15:53:24.690] [debug] [Sequence.cpp:39] Kompute sequence called BEGIN
[2022-08-05 15:53:24.690] [info] [Sequence.cpp:51] Kompute Sequence command now started recording
[2022-08-05 15:53:27.710] [debug] [Manager.cpp:26] [VALIDATION]: Validation - Validation Error: [ VUID-vkBeginCommandBuffer-commandBuffer-00050 ] Object 0: handle = 0x163adcf19e0, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0xcad092000000000d, type = VK_OBJECT_TYPE_COMMAND_POOL; | MessageID = 0xb24f00f5 | Call to vkBeginCommandBuffer() on VkCommandBuffer 0x163adcf19e0[] attempts to implicitly reset cmdBuffer created from VkCommandPool 0xcad092000000000d[] that does NOT have the VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT bit set. The Vulkan spec states: If commandBuffer was allocated from a VkCommandPool which did not have the VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT flag set, commandBuffer must be in the initial state (https://vulkan.lunarg.com/doc/view/1.2.198.1/windows/1.2-extensions/vkspec.html#VUID-vkBeginCommandBuffer-commandBuffer-00050)
...
[2022-08-05 15:53:34.844] [debug] [OpTensorSyncDevice.cpp:42] Kompute OpTensorSyncDevice preEval called
[2022-08-05 15:53:34.845] [debug] [OpAlgoDispatch.cpp:49] Kompute OpAlgoDispatch preEval called
[2022-08-05 15:53:34.846] [debug] [Sequence.cpp:132] Kompute sequence submitting command buffer into compute queue
[2022-08-05 15:53:34.847] [debug] [OpTensorSyncDevice.cpp:48] Kompute OpTensorSyncDevice postEval called
[2022-08-05 15:53:34.848] [debug] [OpAlgoDispatch.cpp:55] Kompute OpAlgoDispatch postSubmit called
[2022-08-05 15:54:24.201] [debug] [kompute/operations/OpAlgoDispatch.hpp:32] Kompute OpAlgoDispatch constructor
[2022-08-05 15:54:24.202] [debug] [Sequence.cpp:86] Kompute Sequence calling clear
[2022-08-05 15:54:24.202] [debug] [Sequence.cpp:279] Kompute Sequence record function started
[2022-08-05 15:54:24.203] [debug] [Sequence.cpp:39] Kompute sequence called BEGIN
[2022-08-05 15:54:24.203] [info] [Sequence.cpp:51] Kompute Sequence command now started recording
[2022-08-05 15:54:27.651] [debug] [Manager.cpp:26] [VALIDATION]: Validation - Validation Error: [ VUID-vkBeginCommandBuffer-commandBuffer-00050 ] Object 0: handle = 0x163adcf19e0, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0xcad092000000000d, type = VK_OBJECT_TYPE_COMMAND_POOL; | MessageID = 0xb24f00f5 | Call to vkBeginCommandBuffer() on VkCommandBuffer 0x163adcf19e0[] attempts to implicitly reset cmdBuffer created from VkCommandPool 0xcad092000000000d[] that does NOT have the VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT bit set. The Vulkan spec states: If commandBuffer was allocated from a VkCommandPool which did not have the VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT flag set, commandBuffer must be in the initial state (https://vulkan.lunarg.com/doc/view/1.2.198.1/windows/1.2-extensions/vkspec.html#VUID-vkBeginCommandBuffer-commandBuffer-00050)
...
[2022-08-05 15:54:33.111] [debug] [OpTensorSyncDevice.cpp:42] Kompute OpTensorSyncDevice preEval called
[2022-08-05 15:54:33.112] [debug] [OpAlgoDispatch.cpp:49] Kompute OpAlgoDispatch preEval called
[2022-08-05 15:54:33.112] [debug] [OpAlgoDispatch.cpp:49] Kompute OpAlgoDispatch preEval called
[2022-08-05 15:54:33.113] [debug] [Sequence.cpp:132] Kompute sequence submitting command buffer into compute queue
[2022-08-05 15:54:33.114] [debug] [OpTensorSyncDevice.cpp:48] Kompute OpTensorSyncDevice postEval called
[2022-08-05 15:54:33.115] [debug] [OpAlgoDispatch.cpp:55] Kompute OpAlgoDispatch postSubmit called
[2022-08-05 15:54:33.115] [debug] [OpAlgoDispatch.cpp:55] Kompute OpAlgoDispatch postSubmit called
[2022-08-05 15:54:58.501] [debug] [OpTensorSyncLocal.cpp:12] Kompute OpTensorSyncLocal constructor with params
[2022-08-05 15:54:58.502] [debug] [Sequence.cpp:86] Kompute Sequence calling clear
[2022-08-05 15:54:58.503] [debug] [Sequence.cpp:279] Kompute Sequence record function started
[2022-08-05 15:54:58.503] [debug] [Sequence.cpp:39] Kompute sequence called BEGIN
[2022-08-05 15:54:58.503] [info] [Sequence.cpp:51] Kompute Sequence command now started recording
[2022-08-05 15:55:00.382] [debug] [Manager.cpp:26] [VALIDATION]: Validation - Validation Error: [ VUID-vkBeginCommandBuffer-commandBuffer-00050 ] Object 0: handle = 0x163adcf19e0, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0xcad092000000000d, type = VK_OBJECT_TYPE_COMMAND_POOL; | MessageID = 0xb24f00f5 | Call to vkBeginCommandBuffer() on VkCommandBuffer 0x163adcf19e0[] attempts to implicitly reset cmdBuffer created from VkCommandPool 0xcad092000000000d[] that does NOT have the VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT bit set. The Vulkan spec states: If commandBuffer was allocated from a VkCommandPool which did not have the VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT flag set, commandBuffer must be in the initial state (https://vulkan.lunarg.com/doc/view/1.2.198.1/windows/1.2-extensions/vkspec.html#VUID-vkBeginCommandBuffer-commandBuffer-00050)
...
[2022-08-05 15:55:00.382] [debug] [Manager.cpp:26] [VALIDATION]: Validation - Validation Error: [ VUID-vkBeginCommandBuffer-commandBuffer-00050 ] Object 0: handle = 0x163adcf19e0, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0xcad092000000000d, type = VK_OBJECT_TYPE_COMMAND_POOL; | MessageID = 0xb24f00f5 | Call to vkBeginCommandBuffer() on VkCommandBuffer 0x163adcf19e0[] attempts to implicitly reset cmdBuffer created from VkCommandPool 0xcad092000000000d[] that does NOT have the VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT bit set. The Vulkan spec states: If commandBuffer was allocated from a VkCommandPool which did not have the VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT flag set, commandBuffer must be in the initial state (https://vulkan.lunarg.com/doc/view/1.2.198.1/windows/1.2-extensions/vkspec.html#VUID-vkBeginCommandBuffer-commandBuffer-00050)
Ok I see, it seems it's a validation error from the validation pipelines, I see what you mean now. Let me try to replicate and I can confirm the results.
@axsaucedo Thanks, I think use vk::CommandPoolCreateFlagBits::eResetCommandBuffer to create commandpool may avoid the validation error