error while finetune with BELLE-LLaMA-EXT-7B

[better629]

finetune using train/training_scripts/single_node/run_LoRA.sh with --model_name_or_path /path/to/decrypted/BELLE-LLaMA-EXT-7B/

But happens with the error

│    130 │   │   util_ops = UtilsBuilder().load()                                                  │
│    131 │   │   self.flatten = util_ops.flatten                                                   │
│    132 │   │   self.unflatten = util_ops.unflatten                                               │
│ ❱  133 │   │   self.dtype = self.optimizer.param_groups[0]['params'][0].dtype                    │
│    134 │   │   self._global_grad_norm = 0.                                                       │
│    135 │   │                                                                                     │
│    136 │   │   self.custom_loss_scaler = False                                                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
IndexError: list index out of range

it seems that can't get a right optimizer inside get_optimizer_grouped_parameters of train/utils/utils.py

[nishiwen1214]

I also got this error, did you solve it?

[better629]

@nishiwen1214 The lastest code(until 20230421 ) maybe can't work well with these pretrains(BELLE-LLaMA-EXT-7B, BELLE-7B-2M, BELLE-LLaMA-7B-2M). You can git checkout 829c10086ac40d7d4fecba39bfea8f6680015d34 to use old version finetune.py to get a start.

[xianghuisun]

finetune using train/training_scripts/single_node/run_LoRA.sh with --model_name_or_path /path/to/decrypted/BELLE-LLaMA-EXT-7B/

But happens with the error

│    130 │   │   util_ops = UtilsBuilder().load()                                                  │
│    131 │   │   self.flatten = util_ops.flatten                                                   │
│    132 │   │   self.unflatten = util_ops.unflatten                                               │
│ ❱  133 │   │   self.dtype = self.optimizer.param_groups[0]['params'][0].dtype                    │
│    134 │   │   self._global_grad_norm = 0.                                                       │
│    135 │   │                                                                                     │
│    136 │   │   self.custom_loss_scaler = False                                                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
IndexError: list index out of range

it seems that can't get a right optimizer inside get_optimizer_grouped_parameters of train/utils/utils.py

We have updated the code, you can experiment based on the latest code.