Phani Srikanth

Sure, here you go. architecture: backbone_dtype: int8 force_embedding_gradients: false gradient_checkpointing: false intermediate_dropout: 0.0 pretrained: true pretrained_weights: '' augmentation: random_parent_probability: 0.0 skip_parent_probability: 0.0 token_mask_probability: 0.0 dataset: add_eos_token_to_answer: true add_eos_token_to_prompt: true answer_column:...

Here are the steps I followed before setting up h2o-llmstudio. 1. I used this AMI from AWS - Deep Learning AMI GPU PyTorch 2.0.1 (Ubuntu 20.04) 20230620 2. Installed Python...

Understood, thanks Max. @psinger, I tried this yesterday and I see the same failure. On Tue, Jun 27, 2023 at 13:15 Philipp Singer ***@***.***> wrote: > @binga could you please...

@psinger -- I am running it via GUI. Happy to do a quick call and debug together if you're available!

Understood. Trying it now.

@psinger - `int4` finetuning works on V100 (32GB) machine. Thank you. `int8` shows the same error described in #185 . This is with `gradient_checkpointing=True`.

Thanks for opening this issue. Does the addition of DQ to 4-bit finetuning need additional testing or just a PR?

As NF4+DQ produces better results (in terms of memory and performance), to keep the interface relatively beginner-friendly, I've added DQ as a default, when int4 finetuning is selected, in this...