RuntimeError:expected scalar type Half but found Float

[felixstander]

I have ran the script of finetune.py on my device, which has two V100 16G gpus, but encounter the problem of RuntimeError: expected scalar type Half but found Float. Does anyone meet the similar problem? Could anyone help me with this? Thanks a lot.

Here are my pacakage version: transformers: 4.28.0.dev0 peft: 0.2.0 bitsandbtyes: 0.37.1 torch: 1.13.1

[deepankar27]

I am also getting same exception. Did you get how to resolve this?

[deepankar27]

add with torch.autocast("cuda"): and it will work.

with torch.autocast("cuda"):
        inputs = tokenizer(prompt, return_tensors="pt")
        input_ids = inputs["input_ids"].to("cuda")

        with torch.no_grad():
            generation_output = model.generate(
                input_ids=input_ids,
                generation_config=generation_config,
                return_dict_in_generate=True,
                output_scores=True,
                max_new_tokens=100
            )

        for s in generation_output.sequences:
            output = tokenizer.decode(s)
            print("Response:", output.split("### Response:")[1].strip())
            return output.split("### Response:")[1].strip()

[elven2016]

the same issue too

[elven2016]

add with torch.autocast("cuda"): and it will work.

with torch.autocast("cuda"):
        inputs = tokenizer(prompt, return_tensors="pt")
        input_ids = inputs["input_ids"].to("cuda")

        with torch.no_grad():
            generation_output = model.generate(
                input_ids=input_ids,
                generation_config=generation_config,
                return_dict_in_generate=True,
                output_scores=True,
                max_new_tokens=100
            )

        for s in generation_output.sequences:
            output = tokenizer.decode(s)
            print("Response:", output.split("### Response:")[1].strip())
            return output.split("### Response:")[1].strip()

where do these code to place? please

[xllau]

I have solved the problem by add "with torch.autocast("cuda"): " before trainer.train(***) in fineturne.py. Good luck!

[itsaadish]

these solutions not working for me

[itsaadish]

Traceback (most recent call last): File "alpaca_fine_tuning.py", line 259, in trainer.train() File "/home/kokil/.local/lib/python3.8/site-packages/transformers/trainer.py", line 1645, in train return inner_training_loop( File "/home/kokil/.local/lib/python3.8/site-packages/transformers/trainer.py", line 1938, in _inner_training_loop tr_loss_step = self.training_step(model, inputs) File "/home/kokil/.local/lib/python3.8/site-packages/transformers/trainer.py", line 2770, in training_step self.accelerator.backward(loss) File "/home/kokil/.local/lib/python3.8/site-packages/accelerate/accelerator.py", line 1819, in backward self.scaler.scale(loss).backward(**kwargs) File "/home/kokil/.local/lib/python3.8/site-packages/torch/_tensor.py", line 487, in backward torch.autograd.backward( File "/home/kokil/.local/lib/python3.8/site-packages/torch/autograd/init.py", line 200, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass File "/home/kokil/.local/lib/python3.8/site-packages/torch/autograd/function.py", line 274, in apply return user_fn(self, *args) File "/home/kokil/.local/lib/python3.8/site-packages/torch/utils/checkpoint.py", line 157, in backward torch.autograd.backward(outputs_with_grad, args_with_grad) File "/home/kokil/.local/lib/python3.8/site-packages/torch/autograd/init.py", line 200, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass File "/home/kokil/.local/lib/python3.8/site-packages/torch/autograd/function.py", line 274, in apply return user_fn(self, *args) File "/home/kokil/.local/lib/python3.8/site-packages/bitsandbytes/autograd/_functions.py", line 456, in backward grad_A = torch.matmul(grad_output, CB).view(ctx.grad_shape).to(ctx.dtype_A) RuntimeError: expected scalar type Half but found Float

[itsaadish]

with torch.autocast("cuda"): trainer.train() @xllau do you mean like this?