torch._dynamo.exc.Unsupported: generator

[anicolson]

System Info

- `Accelerate` version: 1.6.0
- Platform: Linux-5.14.21-150500.55.88-default-x86_64-with-glibc2.31
- `accelerate` bash location: /scratch3/nic261/environments/cxrmate2_virga_rev_a/bin/accelerate
- Python version: 3.12.0
- Numpy version: 1.26.4
- PyTorch version (GPU?): 2.6.0+cu124 (True)
- PyTorch XPU available: False
- PyTorch NPU available: False
- PyTorch MLU available: False
- PyTorch SDAA available: False
- PyTorch MUSA available: False
- System RAM: 2015.38 GB
- GPU type: NVIDIA H100
- `Accelerate` default config:
        Not found

Information

• [ ] The official example scripts
• [x] My own modified scripts

Tasks

• [ ] One of the scripts in the examples/ folder of Accelerate or an officially supported no_trainer script in the examples folder of the transformers repo (such as run_no_trainer_glue.py)
• [x] My own task or dataset (give details below)

Reproduction

import requests
import torch
from accelerate import Accelerator
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

accelerator = Accelerator(mixed_precision="bf16")

model_id = "google/medgemma-4b-pt"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
)
model = accelerator.prepare(model)

image_url = "https://upload.wikimedia.org/wikipedia/commons/c/c8/Chest_Xray_PA_3-8-2010.png"
image = Image.open(
    requests.get(image_url, headers={"User-Agent": "example"}, stream=True).raw
).convert("RGB")

prompt = "<start_of_image> findings:"
inputs = processor(
    text=prompt, images=image, return_tensors="pt"
).to(accelerator.device, dtype=torch.bfloat16)

input_len = inputs["input_ids"].shape[-1]

with torch.no_grad():
    generation = model.generate(**inputs, max_new_tokens=100, do_sample=False)
generation = generation[0][input_len:]

decoded = processor.decode(generation, skip_special_tokens=True)

Expected behavior

I am attempting to train and evaluate this model in Accelerate. I can successfully train it, however, when I evaluate it, I am getting dynamo issues (I don't want to use dynamo).

I have recreated the error in a minimal script (shown above).

CLICK FOR THE ERROR

The error:

Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████| 2/2 [00:08<00:00,  4.17s/it]
The following generation flags are not valid and may be ignored: ['top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
Traceback (most recent call last):
File "_pydevd_sys_monitoring\\_pydevd_sys_monitoring_cython.pyx", line 499, in _pydevd_sys_monitoring_cython._get_code_line_info
KeyError: <code object _is_skip_guard_eval_unsafe_stance at 0x15545bdac120, file "/scratch3/nic261/environments/cxrmate2_virga_rev_a/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 189>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/apps/python/3.12.0/lib/python3.12/runpy.py", line 198, in _run_module_as_main
  return _run_code(code, main_globals, None,
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/apps/python/3.12.0/lib/python3.12/runpy.py", line 88, in _run_code
  exec(code, run_globals)
File "/scratch3/nic261/.vscode_server/virga/.vscode-server/extensions/ms-python.debugpy-2025.8.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 71, in <module>
  cli.main()
File "/scratch3/nic261/.vscode_server/virga/.vscode-server/extensions/ms-python.debugpy-2025.8.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 501, in main
  run()
File "/scratch3/nic261/.vscode_server/virga/.vscode-server/extensions/ms-python.debugpy-2025.8.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 351, in run_file
  runpy.run_path(target, run_name="__main__")
File "/scratch3/nic261/.vscode_server/virga/.vscode-server/extensions/ms-python.debugpy-2025.8.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 310, in run_path
  return _run_module_code(code, init_globals, run_name, pkg_name=pkg_name, script_name=fname)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch3/nic261/.vscode_server/virga/.vscode-server/extensions/ms-python.debugpy-2025.8.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 127, in _run_module_code
  _run_code(code, mod_globals, init_globals, mod_name, mod_spec, pkg_name, script_name)
File "/scratch3/nic261/.vscode_server/virga/.vscode-server/extensions/ms-python.debugpy-2025.8.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 118, in _run_code
  exec(code, run_globals)
File "/datasets/work/hb-mlaifsp-mm/work/repositories/25_cxrmate2/cxrmate2/tests/medgemma_issue.py", line 30, in <module>
  generation = model.generate(**inputs, max_new_tokens=100, do_sample=False)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch3/nic261/environments/cxrmate2_virga_rev_a/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
  return func(*args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^
File "/scratch3/nic261/environments/cxrmate2_virga_rev_a/lib/python3.12/site-packages/transformers/generation/utils.py", line 2597, in generate
  result = self._sample(
           ^^^^^^^^^^^^^
File "/scratch3/nic261/environments/cxrmate2_virga_rev_a/lib/python3.12/site-packages/transformers/generation/utils.py", line 3560, in _sample
  outputs = model_forward(**model_inputs, return_dict=True)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch3/nic261/environments/cxrmate2_virga_rev_a/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 560, in _fn
  _is_skip_guard_eval_unsafe_stance()
File "/scratch3/nic261/environments/cxrmate2_virga_rev_a/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 189, in _is_skip_guard_eval_unsafe_stance
  def _is_skip_guard_eval_unsafe_stance():
  
File "<stringsource>", line 69, in cfunc.to_py.__Pyx_CFunc_893235__29_pydevd_sys_monitoring_cython_object__lParen__etc_to_py_4code_18instruction_offset.wrap
File "_pydevd_sys_monitoring\\_pydevd_sys_monitoring_cython.pyx", line 1702, in _pydevd_sys_monitoring_cython._start_method_event
File "_pydevd_sys_monitoring\\_pydevd_sys_monitoring_cython.pyx", line 570, in _pydevd_sys_monitoring_cython._get_func_code_info
File "_pydevd_sys_monitoring\\_pydevd_sys_monitoring_cython.pyx", line 505, in _pydevd_sys_monitoring_cython._get_code_line_info
File "/scratch3/nic261/environments/cxrmate2_virga_rev_a/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 1380, in __call__
  return self._torchdynamo_orig_callable(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch3/nic261/environments/cxrmate2_virga_rev_a/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 507, in __call__
  unimplemented("generator")
File "/scratch3/nic261/environments/cxrmate2_virga_rev_a/lib/python3.12/site-packages/torch/_dynamo/exc.py", line 317, in unimplemented
  raise Unsupported(msg, case_name=case_name)
torch._dynamo.exc.Unsupported: generator

I have tried doing:

import torch._dynamo
torch._dynamo.disable()

Hopefully, its something basic that I am missing 😅

[anicolson]

Would it be caused by the Hybrid cache? Does it turn on torch dynamo?

[subhash686]

@anicolson dynamo is enabled by default to compile. Can you try this and see if that disables dynamo for you accelerator = Accelerator(mixed_precision="bf16", dynamo_backend="NO") ?

[anicolson]

Thanks for the reply @subhash686

Unfortunately, it didn't work with:

import requests
import torch
from accelerate import Accelerator
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

accelerator = Accelerator(mixed_precision="bf16", dynamo_backend="NO")

model_id = 'google/medgemma-4b-pt'
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
)
model = accelerator.prepare(model)

image_url = "https://upload.wikimedia.org/wikipedia/commons/c/c8/Chest_Xray_PA_3-8-2010.png"
image = Image.open(
    requests.get(image_url, headers={"User-Agent": "example"}, stream=True).raw
).convert("RGB")

prompt = "<start_of_image> findings:"
inputs = processor(
    text=prompt, images=image, return_tensors="pt"
).to(accelerator.device, dtype=torch.bfloat16)

input_len = inputs["input_ids"].shape[-1]

with torch.no_grad():
    generation = model.generate(**inputs, max_new_tokens=100, do_sample=False)
generation = generation[0][input_len:]

decoded = processor.decode(generation, skip_special_tokens=True)

Error

Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████| 2/2 [00:08<00:00,  4.01s/it]
The following generation flags are not valid and may be ignored: ['top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
Traceback (most recent call last):
  File "_pydevd_sys_monitoring\\_pydevd_sys_monitoring_cython.pyx", line 499, in _pydevd_sys_monitoring_cython._get_code_line_info
KeyError: <code object _is_skip_guard_eval_unsafe_stance at 0x15545bdb0120, file "/scratch3/nic261/environments/cxrmate2_virga_rev_a/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 189>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/apps/python/3.12.0/lib/python3.12/runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/apps/python/3.12.0/lib/python3.12/runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "/scratch3/nic261/.vscode_server/virga/.vscode-server/extensions/ms-python.debugpy-2025.8.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 71, in <module>
    cli.main()
  File "/scratch3/nic261/.vscode_server/virga/.vscode-server/extensions/ms-python.debugpy-2025.8.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 501, in main
    run()
  File "/scratch3/nic261/.vscode_server/virga/.vscode-server/extensions/ms-python.debugpy-2025.8.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 351, in run_file
    runpy.run_path(target, run_name="__main__")
  File "/scratch3/nic261/.vscode_server/virga/.vscode-server/extensions/ms-python.debugpy-2025.8.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 310, in run_path
    return _run_module_code(code, init_globals, run_name, pkg_name=pkg_name, script_name=fname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch3/nic261/.vscode_server/virga/.vscode-server/extensions/ms-python.debugpy-2025.8.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 127, in _run_module_code
    _run_code(code, mod_globals, init_globals, mod_name, mod_spec, pkg_name, script_name)
  File "/scratch3/nic261/.vscode_server/virga/.vscode-server/extensions/ms-python.debugpy-2025.8.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 118, in _run_code
    exec(code, run_globals)
  File "/datasets/work/hb-mlaifsp-mm/work/repositories/25_cxrmate2/cxrmate2/tests/medgemma_issue.py", line 30, in <module>
    generation = model.generate(**inputs, max_new_tokens=100, do_sample=False)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch3/nic261/environments/cxrmate2_virga_rev_a/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/scratch3/nic261/environments/cxrmate2_virga_rev_a/lib/python3.12/site-packages/transformers/generation/utils.py", line 2597, in generate
    result = self._sample(
             ^^^^^^^^^^^^^
  File "/scratch3/nic261/environments/cxrmate2_virga_rev_a/lib/python3.12/site-packages/transformers/generation/utils.py", line 3560, in _sample
    outputs = model_forward(**model_inputs, return_dict=True)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch3/nic261/environments/cxrmate2_virga_rev_a/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 560, in _fn
    _is_skip_guard_eval_unsafe_stance()
  File "/scratch3/nic261/environments/cxrmate2_virga_rev_a/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 189, in _is_skip_guard_eval_unsafe_stance
    def _is_skip_guard_eval_unsafe_stance():
    
  File "<stringsource>", line 69, in cfunc.to_py.__Pyx_CFunc_893235__29_pydevd_sys_monitoring_cython_object__lParen__etc_to_py_4code_18instruction_offset.wrap
  File "_pydevd_sys_monitoring\\_pydevd_sys_monitoring_cython.pyx", line 1702, in _pydevd_sys_monitoring_cython._start_method_event
  File "_pydevd_sys_monitoring\\_pydevd_sys_monitoring_cython.pyx", line 570, in _pydevd_sys_monitoring_cython._get_func_code_info
  File "_pydevd_sys_monitoring\\_pydevd_sys_monitoring_cython.pyx", line 505, in _pydevd_sys_monitoring_cython._get_code_line_info
  File "/scratch3/nic261/environments/cxrmate2_virga_rev_a/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 1380, in __call__
    return self._torchdynamo_orig_callable(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch3/nic261/environments/cxrmate2_virga_rev_a/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 507, in __call__
    unimplemented("generator")
  File "/scratch3/nic261/environments/cxrmate2_virga_rev_a/lib/python3.12/site-packages/torch/_dynamo/exc.py", line 317, in unimplemented
    raise Unsupported(msg, case_name=case_name)
torch._dynamo.exc.Unsupported: generator

[LiuYuWei]

Get the same error.

https://github.com/Google-Health/medgemma/issues/13

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.