Got cutlass error: Error Internal at: 214

[syntrive]

Hi, when I run 05_stable_diffusion# python3 src/benchmark.py, there is a error:

pt output: torch.Size([1, 77, 1024]) [gemm_rcr_bias_add_25.cu] Got cutlass error: Error Internal at: 214 [20:21:02] model_interface.cu:221: Error: [gemm_rcr_bias_add_25.cu] Got cutlass error: Error Internal at: 214 Traceback (most recent call last): File "src/benchmark.py", line 312, in benchmark_diffusers() File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1130, in call return self.main(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "src/benchmark.py", line 288, in benchmark_diffusers benchmark_clip( File "src/benchmark.py", line 176, in benchmark_clip exe_module.run_with_tensors(inputs, ys) File "/usr/local/lib/python3.8/dist-packages/aitemplate/compiler/model.py", line 597, in run_with_tensors outputs_ait = self.run( File "/usr/local/lib/python3.8/dist-packages/aitemplate/compiler/model.py", line 495, in run return self._run_impl( File "/usr/local/lib/python3.8/dist-packages/aitemplate/compiler/model.py", line 434, in _run_impl self.DLL.AITemplateModelContainerRun( File "/usr/local/lib/python3.8/dist-packages/aitemplate/compiler/model.py", line 196, in _wrapped_func raise RuntimeError(f"Error in function: {method.name}") RuntimeError: Error in function: AITemplateModelContainerRun

How can I solve it? thank you! env cuda11.7 cutlass3.4.1

[syntrive]

device is A800