peft
peft copied to clipboard
Published
20 hours ago •
huggingface
huggingface
AttributeError: 'float' object has no attribute 'meta'
System Info
peft== 0.17.1 torch== 2.5.1+cu118 transformers==4.57.0 python==3.12.7
Who can help?
I am trying to use LoRA with DINOv3 (so a slightly modified vit-b). However, I am hitting after a random number of iterations this error. It is sadly difficult to reproduce. Maybe someone can hint at what is going on?
Traceback (most recent call last):
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1446, in _call_user_compiler
compiled_fn = compiler_fn(gm, self.example_inputs())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/repro/after_dynamo.py", line 129, in __call__
compiled_gm = compiler_fn(gm, example_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/__init__.py", line 2234, in __call__
return compile_fx(model_, inputs_, config_patches=self.config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1521, in compile_fx
return aot_autograd(
^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/backends/common.py", line 72, in __call__
cg = aot_module_simplified(gm, example_inputs, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 1071, in aot_module_simplified
compiled_fn = dispatch_and_compile()
^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 1056, in dispatch_and_compile
compiled_fn, _ = create_aot_dispatcher_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 522, in create_aot_dispatcher_function
return _create_aot_dispatcher_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 759, in _create_aot_dispatcher_function
compiled_fn, fw_metadata = compiler_fn(
^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 179, in aot_dispatch_base
compiled_fw = compiler(fw_module, updated_flat_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1350, in fw_compiler_base
return _fw_compiler_base(model, example_inputs, is_inference)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1421, in _fw_compiler_base
return inner_compile(
^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 475, in compile_fx_inner
return wrap_compiler_debug(_compile_fx_inner, compiler_name="inductor")(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/repro/after_aot.py", line 85, in debug_wrapper
inner_compiled_fn = compiler_fn(gm, example_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 661, in _compile_fx_inner
compiled_graph = FxGraphCache.load(
^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_inductor/codecache.py", line 1334, in load
compiled_graph = compile_fx_fn(
^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 570, in codegen_and_compile
compiled_graph = fx_codegen_and_compile(gm, example_inputs, **fx_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 789, in fx_codegen_and_compile
_recursive_post_grad_passes(gm, is_inference=is_inference)
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 288, in _recursive_post_grad_passes
post_grad_passes(gm, is_inference)
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_inductor/fx_passes/post_grad.py", line 100, in post_grad_passes
patterns.apply(gm.graph) # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_inductor/pattern_matcher.py", line 1729, in apply
if is_match(m) and entry.extra_check(m):
^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_inductor/fx_passes/quantization.py", line 1448, in fn
scales = match.kwargs["scales"].meta["val"]
^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'float' object has no attribute 'meta'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/k539i/projects/nnunetv2/nnunetv2/inference/predict_from_raw_data.py", line 666, in predict_sliding_window_return_logits
predicted_logits = self._internal_predict_sliding_window_return_logits(data, slicers,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/k539i/projects/nnunetv2/nnunetv2/inference/predict_from_raw_data.py", line 630, in _internal_predict_sliding_window_return_logits
raise e
File "/home/k539i/projects/nnunetv2/nnunetv2/inference/predict_from_raw_data.py", line 609, in _internal_predict_sliding_window_return_logits
prediction = self._internal_maybe_mirror_and_predict(workon)[0].to(results_device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/k539i/projects/nnunetv2/nnunetv2/inference/predict_from_raw_data.py", line 543, in _internal_maybe_mirror_and_predict
prediction = self.network(x)
^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 465, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/k539i/projects/nnunetv2/nnunetv2/training/nnUNetTrainer/variants/curia/nnUNetTrainerCuria.py", line 758, in forward
x = self.backbone(x)
^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/peft/peft_model.py", line 879, in forward
with self._enable_peft_forward_hooks(*args, **kwargs):
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/peft/peft_model.py", line 881, in torch_dynamo_resume_in_forward_at_879
return self.get_base_model()(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 1269, in __call__
return self._torchdynamo_orig_callable(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 1064, in __call__
result = self._inner_convert(
^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 526, in __call__
return _compile(
^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 924, in _compile
guarded_code = compile_inner(code, one_graph, hooks, transform)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 666, in compile_inner
return _compile_inner(code, one_graph, hooks, transform)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_utils_internal.py", line 87, in wrapper_function
return function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 699, in _compile_inner
out_code = transform_code_object(code, transform)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object
transformations(instructions, code_options)
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 219, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 634, in transform
tracer.run()
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/symbolic_convert.py", line 2796, in run
super().run()
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/symbolic_convert.py", line 983, in run
while self.step():
^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/symbolic_convert.py", line 895, in step
self.dispatch_table[inst.opcode](self, inst)
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/symbolic_convert.py", line 2987, in RETURN_VALUE
self._return(inst)
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/symbolic_convert.py", line 2972, in _return
self.output.compile_subgraph(
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1117, in compile_subgraph
self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1369, in compile_and_call_fx_graph
compiled_fn = self.call_user_compiler(gm)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1416, in call_user_compiler
return self._call_user_compiler(gm)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1465, in _call_user_compiler
raise BackendCompilerFailed(self.compiler_fn, e) from e
torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
AttributeError: 'float' object has no attribute 'meta'
Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
You can suppress this exception and fall back to eager by setting:
import torch._dynamo
torch._dynamo.config.suppress_errors = True
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1446, in _call_user_compiler
compiled_fn = compiler_fn(gm, self.example_inputs())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/repro/after_dynamo.py", line 129, in __call__
compiled_gm = compiler_fn(gm, example_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/__init__.py", line 2234, in __call__
return compile_fx(model_, inputs_, config_patches=self.config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1521, in compile_fx
return aot_autograd(
^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/backends/common.py", line 72, in __call__
cg = aot_module_simplified(gm, example_inputs, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 1071, in aot_module_simplified
compiled_fn = dispatch_and_compile()
^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 1056, in dispatch_and_compile
compiled_fn, _ = create_aot_dispatcher_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 522, in create_aot_dispatcher_function
return _create_aot_dispatcher_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 759, in _create_aot_dispatcher_function
compiled_fn, fw_metadata = compiler_fn(
^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 179, in aot_dispatch_base
compiled_fw = compiler(fw_module, updated_flat_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1350, in fw_compiler_base
return _fw_compiler_base(model, example_inputs, is_inference)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1421, in _fw_compiler_base
return inner_compile(
^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 475, in compile_fx_inner
return wrap_compiler_debug(_compile_fx_inner, compiler_name="inductor")(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/repro/after_aot.py", line 85, in debug_wrapper
inner_compiled_fn = compiler_fn(gm, example_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 661, in _compile_fx_inner
compiled_graph = FxGraphCache.load(
^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_inductor/codecache.py", line 1334, in load
compiled_graph = compile_fx_fn(
^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 570, in codegen_and_compile
compiled_graph = fx_codegen_and_compile(gm, example_inputs, **fx_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 789, in fx_codegen_and_compile
_recursive_post_grad_passes(gm, is_inference=is_inference)
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 288, in _recursive_post_grad_passes
post_grad_passes(gm, is_inference)
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_inductor/fx_passes/post_grad.py", line 100, in post_grad_passes
patterns.apply(gm.graph) # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_inductor/pattern_matcher.py", line 1729, in apply
if is_match(m) and entry.extra_check(m):
^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_inductor/fx_passes/quantization.py", line 1448, in fn
scales = match.kwargs["scales"].meta["val"]
^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'float' object has no attribute 'meta'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/bin/nnUNetv2_train", line 8, in <module>
sys.exit(run_training_entry())
^^^^^^^^^^^^^^^^^^^^
File "/home/k539i/projects/nnunetv2/nnunetv2/run/run_training.py", line 292, in run_training_entry
run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights,
File "/home/k539i/projects/nnunetv2/nnunetv2/run/run_training.py", line 229, in run_training
nnunet_trainer.perform_actual_validation(export_validation_probabilities)
File "/home/k539i/projects/nnunetv2/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1289, in perform_actual_validation
prediction = predictor.predict_sliding_window_return_logits(data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/k539i/projects/nnunetv2/nnunetv2/inference/predict_from_raw_data.py", line 672, in predict_sliding_window_return_logits
predicted_logits = self._internal_predict_sliding_window_return_logits(data, slicers, False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/k539i/projects/nnunetv2/nnunetv2/inference/predict_from_raw_data.py", line 630, in _internal_predict_sliding_window_return_logits
raise e
File "/home/k539i/projects/nnunetv2/nnunetv2/inference/predict_from_raw_data.py", line 609, in _internal_predict_sliding_window_return_logits
prediction = self._internal_maybe_mirror_and_predict(workon)[0].to(results_device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/k539i/projects/nnunetv2/nnunetv2/inference/predict_from_raw_data.py", line 543, in _internal_maybe_mirror_and_predict
prediction = self.network(x)
^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 465, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/k539i/projects/nnunetv2/nnunetv2/training/nnUNetTrainer/variants/curia/nnUNetTrainerCuria.py", line 758, in forward
x = self.backbone(x)
^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/peft/peft_model.py", line 879, in forward
with self._enable_peft_forward_hooks(*args, **kwargs):
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/peft/peft_model.py", line 881, in torch_dynamo_resume_in_forward_at_879
return self.get_base_model()(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 1269, in __call__
return self._torchdynamo_orig_callable(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 1064, in __call__
result = self._inner_convert(
^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 526, in __call__
return _compile(
^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 924, in _compile
guarded_code = compile_inner(code, one_graph, hooks, transform)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 666, in compile_inner
return _compile_inner(code, one_graph, hooks, transform)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_utils_internal.py", line 87, in wrapper_function
return function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 699, in _compile_inner
out_code = transform_code_object(code, transform)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object
transformations(instructions, code_options)
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 219, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 634, in transform
tracer.run()
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/symbolic_convert.py", line 2796, in run
super().run()
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/symbolic_convert.py", line 983, in run
while self.step():
^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/symbolic_convert.py", line 895, in step
self.dispatch_table[inst.opcode](self, inst)
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/symbolic_convert.py", line 2987, in RETURN_VALUE
self._return(inst)
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/symbolic_convert.py", line 2972, in _return
self.output.compile_subgraph(
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1117, in compile_subgraph
self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1369, in compile_and_call_fx_graph
compiled_fn = self.call_user_compiler(gm)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1416, in call_user_compiler
return self._call_user_compiler(gm)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dkfz/cluster/gpu/data/OE0441/k539i/miniforge3/envs/nnunetv2/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1465, in _call_user_compiler
raise BackendCompilerFailed(self.compiler_fn, e) from e
torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
AttributeError: 'float' object has no attribute 'meta'
Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
You can suppress this exception and fall back to eager by setting:
import torch._dynamo
torch._dynamo.config.suppress_errors = True
Reproduction
This code is not out-of-the-box runnable but maybe hels for debugging
Lora conf:
lora: train: true r: 8 lora_alpha: 16 lora_dropout: 0.01 target_modules: [attn.qkv, attn.proj, mlp.fc1, mlp.fc2]
Code:
class Pretrained2DSeg3D(nn.Module):
def __init__(self, trainer_conf, backbone_conf, feature_aggregator_conf, head_conf, lora_conf, input_channels, output_channels, **kwargs):
super().__init__()
self.backbone = globals()[backbone_conf["name"]](input_channels=input_channels, **backbone_conf)
self.vit_patch_size = self.backbone.vit_patch_size
self.embed_dim = self.backbone.embed_dim
self.feature_aggregator = globals()[feature_aggregator_conf["name"]](**feature_aggregator_conf, patch_size=trainer_conf["patch_size"], embed_dim=self.embed_dim)
self.head = globals()[head_conf["name"]](**head_conf, embed_dim=self.embed_dim, vit_patch_size=self.vit_patch_size, output_channels=output_channels)
self.ctx = torch.enable_grad()
self.trainer_conf = trainer_conf
self.lora_conf = lora_conf
if lora_conf is not None:
if lora_conf["train"]:
lora_target_modules = lora_conf["target_modules"]
if lora_conf["blocks"] is not None:
lora_target_modules = []
for block in lora_conf["blocks"]:
for m in lora_conf["target_modules"]:
lora_target_modules.append(f"{block}.{m}")
lora_config = LoraConfig(
r=lora_conf["r"], # LoRA rank
lora_alpha=lora_conf["lora_alpha"], # Scaling factor
lora_dropout=lora_conf["lora_dropout"],
target_modules=lora_target_modules
)
self.backbone = get_peft_model(self.backbone, lora_config)
# Freeze all layers except LoRA-adapted ones
for param in self.backbone.parameters():
param.requires_grad = False
for name, param in self.backbone.named_parameters():
if any(sub in name for sub in ["lora"]):
param.requires_grad = True
else:
self.backbone = PeftModel.from_pretrained(self.backbone, Path(trainer_conf["experiment_dir"]) / "lora_ckpt.pth", is_trainable=False)
def save_lora(self):
if self.lora_conf is not None and self.lora_conf["train"]:
self.backbone.save_pretrained(Path(self.trainer_conf["experiment_dir"]) / "lora_ckpt.pth")
Expected behavior
It should not produce the error.
It's really hard to say what's going on here, but from the stack trace, my guess would be an issue with torch.compile itself. Does the training run without torch.compile?
Also, you could try running it without PEFT and with compile to see if it makes any difference. If you don't have enough memory to train without PEFT, try enabling grads on just a single layer for testing purposes.
Hey,
thanks for the reply! Yes it is definetly an issue with compile. According to chatgpt:
This is a known TorchInductor/Dynamo stumble: a quantization fusion pass expects a Tensor “scales” node (with .meta) but sees a Python float, so the matcher crashes during post_grad passes. You’re not even using quantization—this is just an over-eager pattern kicking in around your LoRA-wrapped ViT.
But to be honest I cannot make anything out of this and the proposed solutions are basically to disable torch compile.
The training itself runs without issue without torch compile both with and without PEFT.
I'm not sure if the ChatGPT answer can be trusted here, and even if true, as you mentioned, it's not very actionable. What I would try in your situation:
- Upgrade to a newer PyTorch version
- If this doesn't help, try LoRA dropout of 0 (I don't think 0.01 will make a difference)
- Try to log the batch data that was used when the error occurred: anything suspicious there?
FYI I did not manage to solve the problem but managed to avoid it. I noticed that the error would appear after training from a saved lora checkpoint or when switching to inference mode and doing predictions (no idea why though). I thus made use of the merge_and_unload function to megre the trained lora weights with the model and remove the lora injection. Through this I was able to avoid this error from being triggered.
So this issue can be closed from my side.
Thanks for reporting back. If you haven't tested it, I'd still recommend upgrading PyTorch, which looks like the most promising solution to me.
Sadly this is not recommended for me as we still have V100s in our lab and the latest version of pytorch that still supports these is 2.5.1, which I have.