Vulkan backend crashes with --processors > 1

Open SRHMorris opened this issue 1 year ago • 0 comments

I've compiled main with Vulkan support.

Running on Windows with .\main.exe \path\to\audio.wav -m .\models\ggml-base.bin --processors 2 crashes. This seems to work fine with both the CUDA and Metal backends.

In a debug build I get an assertion failed of _CrtIsValidHeapPointer(block).

When running in a debugger I get various call stacks such as the following two:

nvoglv64.dll!00007ffd6f9b8e9a() (Unknown Source:0)
nvoglv64.dll!00007ffd6fa79ac3() (Unknown Source:0)
nvoglv64.dll!00007ffd6fa79d7d() (Unknown Source:0)
nvoglv64.dll!00007ffd6fa78e8d() (Unknown Source:0)
nvoglv64.dll!00007ffd6fa1c604() (Unknown Source:0)
nvoglv64.dll!00007ffd6fa1c21f() (Unknown Source:0)
vulkan-1.dll!00007ffd94c180cd() (Unknown Source:0)
ggml.dll!vk::DispatchLoaderStatic::vkCmdDispatch(VkCommandBuffer_T * commandBuffer, unsigned int groupCountX, unsigned int groupCountY, unsigned int groupCountZ) Line 1685 (c:\VulkanSDK\1.3.290.0\Include\vulkan\vulkan.hpp:1685)
ggml.dll!vk::CommandBuffer::dispatch<vk::DispatchLoaderStatic>(unsigned int groupCountX, unsigned int groupCountY, unsigned int groupCountZ, const vk::DispatchLoaderStatic & d) Line 5018 (c:\VulkanSDK\1.3.290.0\Include\vulkan\vulkan_funcs.hpp:5018)
ggml.dll!ggml_vk_dispatch_pipeline(ggml_backend_vk_context * ctx, std::shared_ptr<vk_context_struct> & subctx, std::shared_ptr<vk_pipeline_struct> & pipeline, const std::initializer_list<vk::DescriptorBufferInfo> & descriptor_buffer_infos, unsigned __int64 push_constant_size, const void * push_constants, std::array<unsigned int,3> elements) Line 2500 (c:\Users\samue\Development\whisper.cpp\ggml\src\ggml-vulkan.cpp:2500)
ggml.dll!ggml_vk_cpy_to_contiguous(ggml_backend_vk_context * ctx, std::shared_ptr<vk_context_struct> & subctx, std::shared_ptr<vk_pipeline_struct> pipeline, const ggml_tensor * tensor, vk_subbuffer && in, vk_subbuffer && out) Line 3011 (c:\Users\samue\Development\whisper.cpp\ggml\src\ggml-vulkan.cpp:3011)
ggml.dll!ggml_vk_mul_mat_q_f16(ggml_backend_vk_context * ctx, std::shared_ptr<vk_context_struct> & subctx, const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst, bool dryrun) Line 3178 (c:\Users\samue\Development\whisper.cpp\ggml\src\ggml-vulkan.cpp:3178)
ggml.dll!ggml_vk_mul_mat(ggml_backend_vk_context * ctx, std::shared_ptr<vk_context_struct> & subctx, const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst, bool dryrun) Line 3557 (c:\Users\samue\Development\whisper.cpp\ggml\src\ggml-vulkan.cpp:3557)
ggml.dll!ggml_vk_build_graph(ggml_backend_vk_context * ctx, ggml_tensor * node, int node_idx, bool last_node, bool dryrun) Line 5865 (c:\Users\samue\Development\whisper.cpp\ggml\src\ggml-vulkan.cpp:5865)
ggml.dll!ggml_backend_vk_graph_compute(ggml_backend * backend, ggml_cgraph * cgraph) Line 6501 (c:\Users\samue\Development\whisper.cpp\ggml\src\ggml-vulkan.cpp:6501)
ggml.dll!ggml_backend_graph_compute_async(ggml_backend * backend, ggml_cgraph * cgraph) Line 283 (c:\Users\samue\Development\whisper.cpp\ggml\src\ggml-backend.c:283)
ggml.dll!ggml_backend_sched_compute_splits(ggml_backend_sched * sched) Line 1791 (c:\Users\samue\Development\whisper.cpp\ggml\src\ggml-backend.c:1791)
ggml.dll!ggml_backend_sched_graph_compute_async(ggml_backend_sched * sched, ggml_cgraph * graph) Line 1980 (c:\Users\samue\Development\whisper.cpp\ggml\src\ggml-backend.c:1980)
ggml.dll!ggml_backend_sched_graph_compute(ggml_backend_sched * sched, ggml_cgraph * graph) Line 1963 (c:\Users\samue\Development\whisper.cpp\ggml\src\ggml-backend.c:1963)
whisper.dll!ggml_graph_compute_helper(ggml_backend_sched * sched, ggml_cgraph * graph, int n_threads) Line 215 (c:\Users\samue\Development\whisper.cpp\src\whisper.cpp:215)
whisper.dll!whisper_decode_internal(whisper_context & wctx, whisper_state & wstate, const whisper_batch & batch, const int n_threads, bool save_alignment_heads_QKs, bool(*)(void *) abort_callback, void * abort_callback_data) Line 2899 (c:\Users\samue\Development\whisper.cpp\src\whisper.cpp:2899)
whisper.dll!whisper_full_with_state(whisper_context * ctx, whisper_state * state, whisper_full_params params, const float * samples, int n_samples) Line 6087 (c:\Users\samue\Development\whisper.cpp\src\whisper.cpp:6087)
whisper.dll!std::invoke<int (__cdecl*)(whisper_context *,whisper_state *,whisper_full_params,float const *,int),whisper_context *,whisper_state *,whisper_full_params,float const *,int>(int(*)(whisper_context *, whisper_state *, whisper_full_params, const float *, int) && _Obj, whisper_context * && _Arg1, whisper_state * && <_Args2_0>, whisper_full_params && <_Args2_1>, const float * && <_Args2_2>, int && <_Args2_3>) Line 1734 (c:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.41.34120\include\type_traits:1734)
whisper.dll!std::thread::_Invoke<std::tuple<int (__cdecl*)(whisper_context *,whisper_state *,whisper_full_params,float const *,int),whisper_context *,whisper_state *,whisper_full_params,float const *,int>,0,1,2,3,4,5>(void * _RawVals) Line 61 (c:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.41.34120\include\thread:61)
ucrtbased.dll!00007ffd94de3010() (Unknown Source:0)
kernel32.dll!00007ffe4792257d() (Unknown Source:0)
ntdll.dll!00007ffe4996af28() (Unknown Source:0)

nvoglv64.dll!00007ffd6f9b8e9a() (Unknown Source:0)
nvoglv64.dll!00007ffd6fa85abe() (Unknown Source:0)
nvoglv64.dll!00007ffd6fa856f2() (Unknown Source:0)
nvoglv64.dll!00007ffd6fa85e85() (Unknown Source:0)
nvoglv64.dll!00007ffd6fa86de3() (Unknown Source:0)
nvoglv64.dll!00007ffd6fa824ba() (Unknown Source:0)
nvoglv64.dll!00007ffd6fa82bde() (Unknown Source:0)
nvoglv64.dll!00007ffd6fa15e8c() (Unknown Source:0)
vulkan-1.dll!00007ffd944f890b() (Unknown Source:0)
ggml.dll!vk::DispatchLoaderStatic::vkCmdPipelineBarrier(VkCommandBuffer_T * commandBuffer, unsigned int srcStageMask, unsigned int dstStageMask, unsigned int dependencyFlags, unsigned int memoryBarrierCount, const VkMemoryBarrier * pMemoryBarriers, unsigned int bufferMemoryBarrierCount, const VkBufferMemoryBarrier * pBufferMemoryBarriers, unsigned int imageMemoryBarrierCount, const VkImageMemoryBarrier * pImageMemoryBarriers) Line 1840 (c:\VulkanSDK\1.3.290.0\Include\vulkan\vulkan.hpp:1840)
ggml.dll!vk::CommandBuffer::pipelineBarrier<vk::DispatchLoaderStatic>(vk::Flags<enum vk::PipelineStageFlagBits> srcStageMask, vk::Flags<enum vk::PipelineStageFlagBits> dstStageMask, vk::Flags<enum vk::DependencyFlagBits> dependencyFlags, const vk::ArrayProxy<vk::MemoryBarrier const> & memoryBarriers, const vk::ArrayProxy<vk::BufferMemoryBarrier const> & bufferMemoryBarriers, const vk::ArrayProxy<vk::ImageMemoryBarrier const> & imageMemoryBarriers, const vk::DispatchLoaderStatic & d) Line 5548 (c:\VulkanSDK\1.3.290.0\Include\vulkan\vulkan_funcs.hpp:5548)
ggml.dll!ggml_vk_sync_buffers(std::shared_ptr<vk_context_struct> & ctx) Line 1113 (c:\Users\samue\Development\whisper.cpp\ggml\src\ggml-vulkan.cpp:1113)
ggml.dll!ggml_vk_buffer_read_2d_async(std::shared_ptr<vk_context_struct> subctx, std::shared_ptr<vk_buffer_struct> & src, unsigned __int64 offset, void * dst, unsigned __int64 spitch, unsigned __int64 dpitch, unsigned __int64 width, unsigned __int64 height, bool sync_staging) Line 2794 (c:\Users\samue\Development\whisper.cpp\ggml\src\ggml-vulkan.cpp:2794)
ggml.dll!ggml_vk_buffer_read_async(std::shared_ptr<vk_context_struct> subctx, std::shared_ptr<vk_buffer_struct> & src, unsigned __int64 offset, void * dst, unsigned __int64 size, bool sync_staging) Line 2801 (c:\Users\samue\Development\whisper.cpp\ggml\src\ggml-vulkan.cpp:2801)
ggml.dll!ggml_vk_buffer_read(std::shared_ptr<vk_buffer_struct> & src, unsigned __int64 offset, void * dst, unsigned __int64 size) Line 2813 (c:\Users\samue\Development\whisper.cpp\ggml\src\ggml-vulkan.cpp:2813)
ggml.dll!ggml_backend_vk_buffer_get_tensor(ggml_backend_buffer * buffer, const ggml_tensor * tensor, void * data, unsigned __int64 offset, unsigned __int64 size) Line 6191 (c:\Users\samue\Development\whisper.cpp\ggml\src\ggml-vulkan.cpp:6191)
ggml.dll!ggml_backend_tensor_get(const ggml_tensor * tensor, void * data, unsigned __int64 offset, unsigned __int64 size) Line 246 (c:\Users\samue\Development\whisper.cpp\ggml\src\ggml-backend.c:246)
whisper.dll!whisper_decode_internal(whisper_context & wctx, whisper_state & wstate, const whisper_batch & batch, const int n_threads, bool save_alignment_heads_QKs, bool(*)(void *) abort_callback, void * abort_callback_data) Line 2909 (c:\Users\samue\Development\whisper.cpp\src\whisper.cpp:2909)
whisper.dll!whisper_full_with_state(whisper_context * ctx, whisper_state * state, whisper_full_params params, const float * samples, int n_samples) Line 6087 (c:\Users\samue\Development\whisper.cpp\src\whisper.cpp:6087)
whisper.dll!whisper_full_parallel(whisper_context * ctx, whisper_full_params params, const float * samples, int n_samples, int n_processors) Line 6409 (c:\Users\samue\Development\whisper.cpp\src\whisper.cpp:6409)
main.exe!main(int argc, char * * argv) Line 1178 (c:\Users\samue\Development\whisper.cpp\examples\main\main.cpp:1178)
main.exe!invoke_main() Line 79 (d:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:79)
main.exe!__scrt_common_main_seh() Line 288 (d:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)
main.exe!__scrt_common_main() Line 331 (d:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:331)
main.exe!mainCRTStartup(void * __formal) Line 17 (d:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_main.cpp:17)
kernel32.dll!00007ffe4792257d() (Unknown Source:0)
ntdll.dll!00007ffe4996af28() (Unknown Source:0)

I also seem to get random hangs in vkWaitForFences.

If the Vulkan backend doesn't support this then it should be documented somewhere. But it would be unfortunate if this is the case as Vulkan performance is a lot slower than CUDA from my testing (on the same GPU) and doesn't max out the GPU, so potentially allowing multiple processors could improve performance. When using mulitple processors with CUDA I can transcribe 10 minutes of audio in approximately 10 seconds (1 processor is approx 20 seconds). In Vulkan it takes approximately 45 seconds with a single processor.

Sep 11 '24 15:09 SRHMorris