[bug]: AttributeError: 'NoneType' object has no attribute 'cget_col_row_stats' with FLUX Schnell (Quantized)

[Developer-42]

Is there an existing issue for this problem?

• [X] I have searched the existing issues

Operating system

Linux

GPU vendor

AMD (ROCm)

GPU model

AMD Radeon RX 7800 XT

GPU VRAM

16GB

Version number

5.0

Browser

Brave Version 1.70.119 Chromium: 129.0.6668.70 Official Build 64 bit

Python dependencies

{ "accelerate": "0.30.1", "compel": "2.0.2", "cuda": null, "diffusers": "0.27.2", "numpy": "1.26.4", "opencv": "4.9.0.80", "onnx": "1.15.0", "pillow": "10.4.0", "python": "3.11.9", "torch": "2.2.2+rocm5.6", "torchvision": "0.17.2+rocm5.6", "transformers": "4.41.1", "xformers": null }

What happened

I installed FLUX Schnell (Quantized) from the model menu (Starter Models) and then pressed "Invoke" using this model. It gave me this error:

Details

/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/bitsandbytes/autograd/_functions.py:316: UserWarning: MatMul8bitLt: inputs will be cast from torch.bfloat16 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")
[2024-09-26 21:37:08,765]::[InvokeAI]::ERROR --> Error while invoking session 6e5c8ff3-58d8-4559-8c1a-5c700fa2247f, invocation 2c140caa-8845-4a8d-ae39-7893648a5352 (flux_text_encoder): 'NoneType' object has no attribute 'cget_col_row_stats'
[2024-09-26 21:37:08,765]::[InvokeAI]::ERROR --> Traceback (most recent call last):
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/invokeai/app/services/session_processor/session_processor_default.py", line 129, in run_node
    output = invocation.invoke_internal(context=context, services=self._services)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/invokeai/app/invocations/baseinvocation.py", line 290, in invoke_internal
    output = self.invoke(context)
             ^^^^^^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/invokeai/app/invocations/flux_text_encoder.py", line 45, in invoke
    t5_embeddings = self._t5_encode(context)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/invokeai/app/invocations/flux_text_encoder.py", line 69, in _t5_encode
    prompt_embeds = t5_encoder(prompt)
                    ^^^^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/invokeai/backend/flux/modules/conditioner.py", line 28, in forward
    outputs = self.hf_module(
              ^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/transformers/models/t5/modeling_t5.py", line 1972, in forward
    encoder_outputs = self.encoder(
                      ^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/transformers/models/t5/modeling_t5.py", line 1107, in forward
    layer_outputs = layer_module(
                    ^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward
    self_attention_outputs = self.layer[0](
                             ^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/transformers/models/t5/modeling_t5.py", line 594, in forward
    attention_output = self.SelfAttention(
                       ^^^^^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward
    query_states = shape(self.q(hidden_states))  # (batch_size, n_heads, seq_length, dim_per_head)
                         ^^^^^^^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/bitsandbytes/nn/modules.py", line 817, in forward
    out = bnb.matmul(x, self.weight, bias=self.bias, state=self.state)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/bitsandbytes/autograd/_functions.py", line 556, in matmul
    return MatMul8bitLt.apply(A, B, out, bias, state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/torch/autograd/function.py", line 553, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/bitsandbytes/autograd/_functions.py", line 321, in forward
    CA, CAt, SCA, SCAt, coo_tensorA = F.double_quant(A.to(torch.float16), threshold=state.threshold)
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/bitsandbytes/functional.py", line 2523, in double_quant
    row_stats, col_stats, nnz_row_ptr = get_colrow_absmax(A, threshold=threshold)
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/x/ai/invokeAI/.venv/lib/python3.11/site-packages/bitsandbytes/functional.py", line 2422, in get_colrow_absmax
    lib.cget_col_row_stats(ptrA, ptrRowStats, ptrColStats, ptrNnzrows, ct.c_float(threshold), rows, cols)
    ^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'cget_col_row_stats'

What you expected to happen

That it generates an image like in the official 5.0 update video.

Additional context

Before running InvokeAI, I always need to run export HSA_OVERRIDE_GFX_VERSION=11.0.0 but then I can generate images with other models.

[Developer-42]

I can confirm that using the "FLUX Text to Image" workflow like shown here results in the same error.

[BROsandr]

Same with AMD Radeon RX 7600

[TimCabbage]

same with 7900xtx

[hchasens]

same with 7900xtx

same

[Yogomirius]

Got the same problem with 7900GRE

[PixelGames987]

same with 7900GRE

[hchasens]

Has anyone found a fix

[PixelGames987]

The same thing happenes with any flux model

[vectorsigma]

I have the same problems. The tracebacks I see seem to indicate that the python library can't actually instantiate a python object from the model. (lot of the errors are of the form "NoneType object has no attribute ...") The errors happen within 1-4 seconds after clicking invoke. If I change the model and the various supporting bits to use flux.1 dev non-quantized, those errors go away, indicating a problem somewhere in the interplay between the pytorch and quantization modules.

This happens on v5.5.0 installed via pip in a venv, using rocm 6.1 (6.2 apparently breaks invokeAI), pytorch 2.4.2 (newer 2.5.x branch also breaks invokeAI) and python 3.11 in Fedora 40. I can reproduce the issue every time, and quite happy to help debug.

There's an issue with bitsandbytes on startup indicating that bitsandbytes has trouble with some model formats, it might be a red herring though. The error is something like "CUDA is available but not working, go find your CUDA libraries."