taichi
taichi copied to clipboard
Published
20 hours ago •
taichi-dev
taichi-dev
Code might crash with Vulkan backend
I'm running a LBM code in windows with Vulkan backend, it crashed after some several steps. This code can run very well under CPU backend and in Linux with CUDA backend.
I noticed that the memory usage seems continue increasing when using Vulkan backend when I run this code https://github.com/yjhp1016/taichi_LBM3D/blob/main/Single_phase/lbm_solver_3d_cavity.py But I don't have such issue in CPU or CUDA on Linux. The code can run very well for long time.
The error I got is below, if you need any other information, please feel free to let me know
(base) C:\Users\J0424810\Downloads\Work\taichi_LBM3D\Single_phase>C:/Users/J0424810/AppData/Local/Continuum/anaconda3/python.exe c:/Users/J0424810/Downloads/Work/taichi_LBM3D/Single_phase/lbm_solver_3d_cavity.py
[Taichi] version 1.1.0, llvm 10.0.0, commit f5bb6464, win, python 3.7.4
[W 08/18/22 10:12:10.879 11468] [D:/a/taichi/taichi/taichi/rhi/cuda/cuda_driver.cpp:CUDADriver@39] CUDA driver not found.
[Taichi] Starting on arch=vulkan
[I 08/18/22 10:12:11.305 11468] [D:/a/taichi/taichi/taichi/rhi/vulkan/vulkan_device_creator.cpp:pick_physical_device@372] Found Vulkan Device 0 (Intel(R) UHD Graphics 620)
[I 08/18/22 10:12:11.306 11468] [D:/a/taichi/taichi/taichi/rhi/vulkan/vulkan_device_creator.cpp:pick_physical_device@372] Found Vulkan Device 1 (Intel(R) UHD Graphics 620)
[I 08/18/22 10:12:11.312 11468] [D:/a/taichi/taichi/taichi/rhi/vulkan/vulkan_device_creator.cpp:create_logical_device@440] Vulkan Device "Intel(R) UHD Graphics 620" supports Vulkan 0 version 1.2.203
UserWarning: Taichi matrices/vectors with 19x19 > 32 entries are not suggested. Matrices/vectors will be automatically unrolled at compile-time for performance. So the compilation time could be extremely long if the matrix size is too big. You may use a field to store a large matrix like this, e.g.:
x = ti.field(ti.f32, (19, 19)).
See https://docs.taichi-lang.org/docs/field#matrix-size for more details.
File "c:/Users/J0424810/Downloads/Work/taichi_LBM3D/Single_phase/lbm_solver_3d_cavity.py", line 83, in <module>
M[None] = ti.Matrix(M_np)
UserWarning: Taichi matrices/vectors with 19x19 > 32 entries are not suggested. Matrices/vectors will be automatically unrolled at compile-time for performance. So the compilation time could be extremely long if the matrix size is too big. You may use a field to store a large matrix like this, e.g.:
x = ti.field(ti.f32, (19, 19)).
See https://docs.taichi-lang.org/docs/field#matrix-size for more details.
File "c:/Users/J0424810/Downloads/Work/taichi_LBM3D/Single_phase/lbm_solver_3d_cavity.py", line 83, in <module>
M[None] = ti.Matrix(M_np)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\util.py", line 297, in wrapped
return func(*args, **kwargs)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\matrix.py", line 1658, in __setitem__
self[key]._set_entries(value)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\util.py", line 297, in wrapped
return func(*args, **kwargs)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\matrix.py", line 1666, in __getitem__
for i in range(self.n)])
UserWarning: Taichi matrices/vectors with 19x19 > 32 entries are not suggested. Matrices/vectors will be automatically unrolled at compile-time for performance. So the compilation time could be extremely long if the matrix size is too big. You may use a field to store a large matrix like this, e.g.:
x = ti.field(ti.f32, (19, 19)).
See https://docs.taichi-lang.org/docs/field#matrix-size for more details.
File "c:/Users/J0424810/Downloads/Work/taichi_LBM3D/Single_phase/lbm_solver_3d_cavity.py", line 84, in <module>
inv_M[None] = ti.Matrix(inv_M_np)
UserWarning: Taichi matrices/vectors with 19x19 > 32 entries are not suggested. Matrices/vectors will be automatically unrolled at compile-time for performance. So the compilation time could be extremely long if the matrix size is too big. You may use a field to store a large matrix like this, e.g.:
x = ti.field(ti.f32, (19, 19)).
See https://docs.taichi-lang.org/docs/field#matrix-size for more details.
File "c:/Users/J0424810/Downloads/Work/taichi_LBM3D/Single_phase/lbm_solver_3d_cavity.py", line 84, in <module>
inv_M[None] = ti.Matrix(inv_M_np)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\util.py", line 297, in wrapped
return func(*args, **kwargs)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\matrix.py", line 1658, in __setitem__
self[key]._set_entries(value)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\util.py", line 297, in wrapped
return func(*args, **kwargs)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\matrix.py", line 1666, in __getitem__
for i in range(self.n)])
[W 08/18/22 10:12:19.939 11468] [D:/a/taichi/taichi/taichi/transforms/type_check.cpp:type_check_store@36] [$12684] Global store may lose precision: i32 <- f64
On line 121 of file "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\_kernels.py", in ext_arr_to_tensor:
tensor[I] = arr[I]
UserWarning: Taichi matrices/vectors with 19x19 > 32 entries are not suggested. Matrices/vectors will be automatically unrolled at compile-time for performance. So the compilation time could be extremely long if the matrix size is too big. You may use a field to store a large matrix like this, e.g.:
x = ti.field(ti.f32, (19, 19)).
See https://docs.taichi-lang.org/docs/field#matrix-size for more details.
File "c:/Users/J0424810/Downloads/Work/taichi_LBM3D/Single_phase/lbm_solver_3d_cavity.py", line 283, in <module>
init()
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\kernel_impl.py", line 918, in wrapped
return primal(*args, **kwargs)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\kernel_impl.py", line 844, in __call__
key = self.ensure_compiled(*args)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\kernel_impl.py", line 819, in ensure_compiled
self.materialize(key=key, args=args, arg_features=arg_features)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\kernel_impl.py", line 543, in materialize
taichi_ast_generator, kernel_name, self.autodiff_mode)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\kernel_impl.py", line 532, in taichi_ast_generator
transform_tree(tree, ctx)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\transform.py", line 6, in transform_tree
ASTTransformer()(ctx, tree)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer_utils.py", line 24, in __call__
return method(ctx, node)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer.py", line 644, in build_Module
build_stmt(ctx, stmt)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer_utils.py", line 24, in __call__
return method(ctx, node)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer.py", line 592, in build_FunctionDef
build_stmts(ctx, node.body)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer.py", line 1311, in build_stmts
build_stmt(ctx, stmt)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer_utils.py", line 24, in __call__
return method(ctx, node)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer.py", line 92, in build_Assign
build_stmt(ctx, node.value)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer_utils.py", line 24, in __call__
return method(ctx, node)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer.py", line 497, in build_Call
node.ptr = func(*args, **keywords)
UserWarning: Taichi matrices/vectors with 19x19 > 32 entries are not suggested. Matrices/vectors will be automatically unrolled at compile-time for performance. So the compilation time could be extremely long if the matrix size is too big. You may use a field to store a large matrix like this, e.g.:
x = ti.field(ti.f32, (19, 19)).
See https://docs.taichi-lang.org/docs/field#matrix-size for more details.
File "c:/Users/J0424810/Downloads/Work/taichi_LBM3D/Single_phase/lbm_solver_3d_cavity.py", line 283, in <module>
init()
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\kernel_impl.py", line 918, in wrapped
return primal(*args, **kwargs)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\kernel_impl.py", line 844, in __call__
key = self.ensure_compiled(*args)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\kernel_impl.py", line 819, in ensure_compiled
self.materialize(key=key, args=args, arg_features=arg_features)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\kernel_impl.py", line 543, in materialize
taichi_ast_generator, kernel_name, self.autodiff_mode)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\kernel_impl.py", line 532, in taichi_ast_generator
transform_tree(tree, ctx)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\transform.py", line 6, in transform_tree
ASTTransformer()(ctx, tree)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer_utils.py", line 24, in __call__
return method(ctx, node)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer.py", line 644, in build_Module
build_stmt(ctx, stmt)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer_utils.py", line 24, in __call__
return method(ctx, node)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer.py", line 592, in build_FunctionDef
build_stmts(ctx, node.body)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer.py", line 1311, in build_stmts
build_stmt(ctx, stmt)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer_utils.py", line 24, in __call__
return method(ctx, node)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer.py", line 101, in build_Assign
node.value.ptr)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\util.py", line 287, in wrapped
return func(*args, **kwargs)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\impl.py", line 51, in expr_init
return Matrix(rhs.to_list())
UserWarning: Taichi matrices/vectors with 19x19 > 32 entries are not suggested. Matrices/vectors will be automatically unrolled at compile-time for performance. So the compilation time could be extremely long if the matrix size is too big. You may use a field to store a large matrix like this, e.g.:
x = ti.field(ti.f32, (19, 19)).
See https://docs.taichi-lang.org/docs/field#matrix-size for more details.
File "c:/Users/J0424810/Downloads/Work/taichi_LBM3D/Single_phase/lbm_solver_3d_cavity.py", line 283, in <module>
init()
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\kernel_impl.py", line 918, in wrapped
return primal(*args, **kwargs)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\kernel_impl.py", line 844, in __call__
key = self.ensure_compiled(*args)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\kernel_impl.py", line 819, in ensure_compiled
self.materialize(key=key, args=args, arg_features=arg_features)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\kernel_impl.py", line 543, in materialize
taichi_ast_generator, kernel_name, self.autodiff_mode)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\kernel_impl.py", line 532, in taichi_ast_generator
transform_tree(tree, ctx)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\transform.py", line 6, in transform_tree
ASTTransformer()(ctx, tree)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer_utils.py", line 24, in __call__
return method(ctx, node)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer.py", line 644, in build_Module
build_stmt(ctx, stmt)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer_utils.py", line 24, in __call__
return method(ctx, node)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer.py", line 592, in build_FunctionDef
build_stmts(ctx, node.body)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer.py", line 1311, in build_stmts
build_stmt(ctx, stmt)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer_utils.py", line 24, in __call__
return method(ctx, node)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer.py", line 92, in build_Assign
build_stmt(ctx, node.value)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer_utils.py", line 24, in __call__
return method(ctx, node)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer.py", line 497, in build_Call
node.ptr = func(*args, **keywords)
UserWarning: Taichi matrices/vectors with 19x19 > 32 entries are not suggested. Matrices/vectors will be automatically unrolled at compile-time for performance. So the compilation time could be extremely long if the matrix size is too big. You may use a field to store a large matrix like this, e.g.:
x = ti.field(ti.f32, (19, 19)).
See https://docs.taichi-lang.org/docs/field#matrix-size for more details.
File "c:/Users/J0424810/Downloads/Work/taichi_LBM3D/Single_phase/lbm_solver_3d_cavity.py", line 283, in <module>
init()
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\kernel_impl.py", line 918, in wrapped
return primal(*args, **kwargs)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\kernel_impl.py", line 844, in __call__
key = self.ensure_compiled(*args)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\kernel_impl.py", line 819, in ensure_compiled
self.materialize(key=key, args=args, arg_features=arg_features)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\kernel_impl.py", line 543, in materialize
taichi_ast_generator, kernel_name, self.autodiff_mode)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\kernel_impl.py", line 532, in taichi_ast_generator
transform_tree(tree, ctx)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\transform.py", line 6, in transform_tree
ASTTransformer()(ctx, tree)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer_utils.py", line 24, in __call__
return method(ctx, node)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer.py", line 644, in build_Module
build_stmt(ctx, stmt)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer_utils.py", line 24, in __call__
return method(ctx, node)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer.py", line 592, in build_FunctionDef
build_stmts(ctx, node.body)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer.py", line 1311, in build_stmts
build_stmt(ctx, stmt)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer_utils.py", line 24, in __call__
return method(ctx, node)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\ast\ast_transformer.py", line 101, in build_Assign
node.value.ptr)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\util.py", line 287, in wrapped
return func(*args, **kwargs)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\impl.py", line 51, in expr_init
return Matrix(rhs.to_list())
[0.0, 0.0, 0.1]
----------Time between two outputs is 0h 0m 59s; elapsed time is 0h 0m 59s----------------------
The 0th iteration, Max Force = 10.000000, force_scale = 10.000000
----------Time between two outputs is 0h 0m 1s; elapsed time is 0h 1m 0s----------------------
The 1000th iteration, Max Force = 10.000000, force_scale = 10.000000
----------Time between two outputs is 0h 0m 2s; elapsed time is 0h 1m 2s----------------------
The 2000th iteration, Max Force = 10.000000, force_scale = 10.000000
----------Time between two outputs is 0h 0m 3s; elapsed time is 0h 1m 6s----------------------
The 3000th iteration, Max Force = 10.000000, force_scale = 10.000000
----------Time between two outputs is 0h 0m 14s; elapsed time is 0h 1m 20s----------------------
The 4000th iteration, Max Force = 10.000000, force_scale = 10.000000
----------Time between two outputs is 0h 0m 21s; elapsed time is 0h 1m 41s----------------------
The 5000th iteration, Max Force = 10.000000, force_scale = 10.000000
----------Time between two outputs is 0h 0m 24s; elapsed time is 0h 2m 6s----------------------
The 6000th iteration, Max Force = 10.000000, force_scale = 10.000000
[E 08/18/22 10:14:40.804 11468] [D:/a/taichi/taichi/taichi/rhi/vulkan/vulkan_device.cpp:submit@1697] Vulkan Error : -4 : failed to submit command buffer
Traceback (most recent call last):
File "c:/Users/J0424810/Downloads/Work/taichi_LBM3D/Single_phase/lbm_solver_3d_cavity.py", line 287, in <module>
colission()
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\kernel_impl.py", line 918, in wrapped
return primal(*args, **kwargs)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\kernel_impl.py", line 845, in __call__
return self.runtime.compiled_functions[key](*args)
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\kernel_impl.py", line 777, in func__
raise e from None
File "C:\Users\J0424810\AppData\Local\Continuum\anaconda3\lib\site-packages\taichi\lang\kernel_impl.py", line 774, in func__
t_kernel(launch_ctx)
RuntimeError: [D:/a/taichi/taichi/taichi/rhi/vulkan/vulkan_device.cpp:submit@1697] Vulkan Error : -4 : failed to submit command buffer
[E 08/18/22 10:14:41.461 11468] [D:/a/taichi/taichi/taichi/rhi/vulkan/vulkan_device.cpp:submit@1697] Vulkan Error : -4 : failed to submit command buffer
It would take some time to investigate this problem. Could you use the CUDA backend for now?
Thanks for your feedback, no problem, take your time, I can use CUDA for now :-)
Kind regards,
Jianhui
On Fri, 26 Aug 2022 at 08:16, PENGUINLIONG @.***> wrote:
It would take some time to investigate this problem. Could you use the CUDA backend for now?
— Reply to this email directly, view it on GitHub https://github.com/taichi-dev/taichi/issues/5821#issuecomment-1228140838, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJEDKQHQODJZ7R55KJ2ONM3V3BVMJANCNFSM564RFDXQ . You are receiving this because you authored the thread.Message ID: @.***>