RuntimeError: Expected is_sm80 to be true, but got false.

[Qiang-HU]

╭───────────────────────── Traceback (most recent call last) ──────────────────────────╮ │ /Project/lq_ChatGLM2-6B/LLM-Tuning/chatglm2_lora_tuning.py:172 in │ │ │ │ 169 │ │ 170 │ │ 171 if name == "main": │ │ ❱ 172 │ main() │ │ 173 │ │ │ │ /Project/lq_ChatGLM2-6B/LLM-Tuning/chatglm2_lora_tuning.py:165 in main │ │ │ │ 162 │ │ callbacks=[TensorBoardCallback(writer)], │ │ 163 │ │ data_collator=data_collator │ │ 164 │ ) │ │ ❱ 165 │ trainer.train() │ │ 166 │ writer.close() │ │ 167 │ # save model │ │ 168 │ model.save_pretrained(training_args.output_dir) │ │ │ │ /usr/local/python3.8/lib/python3.8/site-packages/transformers/trainer.py:1662 in │ │ train │ │ │ │ 1659 │ │ inner_training_loop = find_executable_batch_size( │ │ 1660 │ │ │ self._inner_training_loop, self._train_batch_size, args.auto_find │ │ 1661 │ │ ) │ │ ❱ 1662 │ │ return inner_training_loop( │ │ 1663 │ │ │ args=args, │ │ 1664 │ │ │ resume_from_checkpoint=resume_from_checkpoint, │ │ 1665 │ │ │ trial=trial, │ │ │ │ /usr/local/python3.8/lib/python3.8/site-packages/transformers/trainer.py:1929 in │ │ _inner_training_loop │ │ │ │ 1926 │ │ │ │ │ with model.no_sync(): │ │ 1927 │ │ │ │ │ │ tr_loss_step = self.training_step(model, inputs) │ │ 1928 │ │ │ │ else: │ │ ❱ 1929 │ │ │ │ │ tr_loss_step = self.training_step(model, inputs) │ │ 1930 │ │ │ │ │ │ 1931 │ │ │ │ if ( │ │ 1932 │ │ │ │ │ args.logging_nan_inf_filter │ │ │ │ /usr/local/python3.8/lib/python3.8/site-packages/transformers/trainer.py:2709 in │ │ training_step │ │ │ │ 2706 │ │ │ loss = loss / self.args.gradient_accumulation_steps │ │ 2707 │ │ │ │ 2708 │ │ if self.do_grad_scaling: │ │ ❱ 2709 │ │ │ self.scaler.scale(loss).backward() │ │ 2710 │ │ elif self.use_apex: │ │ 2711 │ │ │ with amp.scale_loss(loss, self.optimizer) as scaled_loss: │ │ 2712 │ │ │ │ scaled_loss.backward() │ │ │ │ /usr/local/python3.8/lib/python3.8/site-packages/torch/_tensor.py:487 in backward │ │ │ │ 484 │ │ │ │ create_graph=create_graph, │ │ 485 │ │ │ │ inputs=inputs, │ │ 486 │ │ │ ) │ │ ❱ 487 │ │ torch.autograd.backward( │ │ 488 │ │ │ self, gradient, retain_graph, create_graph, inputs=inputs │ │ 489 │ │ ) │ │ 490 │ │ │ │ /usr/local/python3.8/lib/python3.8/site-packages/torch/autograd/init.py:200 in │ │ backward │ │ │ │ 197 │ # The reason we repeat same the comment below is that │ │ 198 │ # some Python versions print out the first line of a multi-line function │ │ 199 │ # calls in the traceback and some print out the last line │ │ ❱ 200 │ Variable.execution_engine.run_backward( # Calls into the C++ engine to r │ │ 201 │ │ tensors, grad_tensors, retain_graph, create_graph, inputs, │ │ 202 │ │ allow_unreachable=True, accumulate_grad=True) # Calls into the C++ en │ │ 203 │ │ │ │ /usr/local/python3.8/lib/python3.8/site-packages/torch/autograd/function.py:274 in │ │ apply │ │ │ │ 271 │ │ │ │ │ │ │ "Function is not allowed. You should only imple │ │ 272 │ │ │ │ │ │ │ "of them.") │ │ 273 │ │ user_fn = vjp_fn if vjp_fn is not Function.vjp else backward_fn │ │ ❱ 274 │ │ return user_fn(self, *args) │ │ 275 │ │ │ 276 │ def apply_jvp(self, *args): │ │ 277 │ │ # _forward_cls is defined by derived class │ │ │ │ /usr/local/python3.8/lib/python3.8/site-packages/torch/utils/checkpoint.py:157 in │ │ backward │ │ │ │ 154 │ │ │ raise RuntimeError( │ │ 155 │ │ │ │ "none of output has requires_grad=True," │ │ 156 │ │ │ │ " this checkpoint() is not necessary") │ │ ❱ 157 │ │ torch.autograd.backward(outputs_with_grad, args_with_grad) │ │ 158 │ │ grads = tuple(inp.grad if isinstance(inp, torch.Tensor) else None │ │ 159 │ │ │ │ │ for inp in detached_inputs) │ │ 160 │ │ │ │ /usr/local/python3.8/lib/python3.8/site-packages/torch/autograd/init.py:200 in │ │ backward │ │ │ │ 197 │ # The reason we repeat same the comment below is that │ │ 198 │ # some Python versions print out the first line of a multi-line function │ │ 199 │ # calls in the traceback and some print out the last line │ │ ❱ 200 │ Variable.execution_engine.run_backward( # Calls into the C++ engine to r │ │ 201 │ │ tensors, grad_tensors, retain_graph, create_graph, inputs, │ │ 202 │ │ allow_unreachable=True, accumulate_grad=True) # Calls into the C++ en │ │ 203 │ ╰──────────────────────────────────────────────────────────────────────────────────────╯ RuntimeError: Expected is_sm80 to be true, but got false. (Could this error message be improved? If so, please report an enhancement request to PyTorch.)

这个报错是为什么？

[beyondguo]

pytorch版本？

[Qiang-HU]

对，先用的torch2.0版本的，后面降到1.13.1版本就行了

[beyondguo]

奇怪，但我使用的torch2.0

[natureLanguageQing]

遇到了