CUBLAS_STATUS_ALLOC_FAILED error

[kjhenner]

I'm getting an error from the text transformer component when trying to run the CLIP guided generation. Any ideas as to how I might approach debugging here?

CUDA_LAUNCH_BLOCKING=1 python sample_clip_guided.py "depth map image of intricate floral pattern" --config configs/config_256x256_depth.json --checkpoint depth_00100000.pth --batch-size 1 --clip-model RN50x4 2> error.txt

  File "/home/kevin/src/k-diffusion/sample_clip_guided.py", line 133, in <module>
    main()
  File "/home/kevin/src/k-diffusion/sample_clip_guided.py", line 99, in main
    target_embed = F.normalize(clip_model.encode_text(clip.tokenize(args.prompt, truncate=True).to(device)).float())
  File "/home/kevin/anaconda3/envs/k-diffusion/lib/python3.10/site-packages/clip/model.py", line 348, in encode_text
    x = self.transformer(x)
  File "/home/kevin/anaconda3/envs/k-diffusion/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/kevin/anaconda3/envs/k-diffusion/lib/python3.10/site-packages/clip/model.py", line 203, in forward
    return self.resblocks(x)
  File "/home/kevin/anaconda3/envs/k-diffusion/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/kevin/anaconda3/envs/k-diffusion/lib/python3.10/site-packages/torch/nn/modules/container.py", line 139, in forward
    input = module(input)
  File "/home/kevin/anaconda3/envs/k-diffusion/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/kevin/anaconda3/envs/k-diffusion/lib/python3.10/site-packages/clip/model.py", line 190, in forward
    x = x + self.attention(self.ln_1(x))
  File "/home/kevin/anaconda3/envs/k-diffusion/lib/python3.10/site-packages/clip/model.py", line 187, in attention
    return self.attn(x, x, x, need_weights=False, attn_mask=self.attn_mask)[0]
  File "/home/kevin/anaconda3/envs/k-diffusion/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/kevin/anaconda3/envs/k-diffusion/lib/python3.10/site-packages/torch/nn/modules/activation.py", line 1153, in forward
    attn_output, attn_output_weights = F.multi_head_attention_forward(
  File "/home/kevin/anaconda3/envs/k-diffusion/lib/python3.10/site-packages/torch/nn/functional.py", line 5066, in multi_head_attention_forward
    q, k, v = _in_projection_packed(query, key, value, in_proj_weight, in_proj_bias)
  File "/home/kevin/anaconda3/envs/k-diffusion/lib/python3.10/site-packages/torch/nn/functional.py", line 4745, in _in_projection_packed
    return linear(q, w, b).chunk(3, dim=-1)
RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)````

[kjhenner]

I switched to a 3090 and followed the process here (https://github.com/pytorch/pytorch/issues/49161) to resolve some issues with that. Seems to be working fine there, so I imagine it's some pytorch/cuda/driver match issue. I'll leave this open for the moment in the off chance it's useful information, but feel free to close!