Anton Bacaj comments

Results 62 comments of


                                            Anton Bacaj

fine tuning the updated Phi-2 with flash-attn-2 produces very high loss > 2

Ok so I remove the explicit `torch_dtype` following the comments in your link. The loss is still very high with flash-attn-2 using [phi-2](https://huggingface.co/microsoft/phi-2) model

fine tuning the updated Phi-2 with flash-attn-2 produces very high loss > 2

I use `padding_side="left"`. Here is how the loss goes with and without FA2 (green line has FA2) using phi-2: ![image](https://github.com/huggingface/transformers/assets/7272343/2c2ac3da-d685-44d2-a919-164e40300ee8)

fine tuning the updated Phi-2 with flash-attn-2 produces very high loss > 2

FWIW changing padding side doesn't do anything to the loss, it's the same

fine tuning the updated Phi-2 with flash-attn-2 produces very high loss > 2

``` adafactor=False, adam_beta1=0.9, adam_beta2=0.95, adam_epsilon=1e-08, auto_find_batch_size=False, bf16=True, bf16_full_eval=False, cache_dir=None, data_seed=None, dataloader_drop_last=False, dataloader_num_workers=0, dataloader_persistent_workers=False, dataloader_pin_memory=True, ddp_backend=None, ddp_broadcast_buffers=None, ddp_bucket_cap_mb=None, ddp_find_unused_parameters=None, ddp_timeout=1800, debug=[], deepspeed=src/configs/deepspeed_2_config.json, disable_tqdm=False, dispatch_batches=None, do_eval=False, do_predict=False, do_train=True, eval_accumulation_steps=None, eval_delay=0, eval_steps=0.0, evaluation_strategy=no,...

fine tuning the updated Phi-2 with flash-attn-2 produces very high loss > 2

FWIW - the model still comes out significantly worse using FA2. If anyone wants to fine-tune this model, I recommend you use it without FA2 currently. Running code benchmarks with...

fine tuning the updated Phi-2 with flash-attn-2 produces very high loss > 2

I do another two runs using FA2 and without FA2 (blue line). Testing the models out using vLLM, the model without FA2 (blue line) scores 58% on humaneval, the model...

Can't resolve symbol io.flutter.plugin dependency

``` PS C:\Users\Anton> flutter doctor -v [√] Flutter (Channel master, v0.5.8-pre.79, on Microsoft Windows [Version 10.0.17134.48], locale en-US) • Flutter version 0.5.8-pre.79 at C:\Users\Anton\Documents\flutter • Framework revision bb338ef2c6 (7 days...