EliverQ

Ingredient : Task Description Prompt content: Make your prompt as detailed as possible, e.g., "Summarize the article into a short paragraph within 50 words. The major storyline and conclusion should...

We welcome everyone to provide us with more relevant tips in the form of issues. After selection, we will regularly update them on GitHub and indicate the source. If you...

As we mention in the survey, we only include LLMs (larger than 10B) with publicly reported evaluation results in Figure 1. Excluding models with papers (because formal evaluation results are...

Evaluation settings of INSTRUCTOR

11

Hello! I have a very puzzling question that I would like to ask. Since your model is fine-tuned with instructions, why not use instructions during benchmark evaluations (e.g. MTEB)?

Reproduction of training INSTRUCTOR

4

Hello! I must say, Instructor is truly an amazing project, and I'm eager to replicate your training process. Nevertheless, despite following your training settings, I'm unable to achieve comparable performance...

xformers error when fine-tuning open_llama_3B with memory_efficient_attention

5

Hi, I feel confused about this bug when using memory_efficient_attention. It seems that the embed per head you choose can't match with xformers? ``` NotImplementedError: No operator found for `memory_efficient_attention_forward`...

请问paddle中是否有类似gradient_checkpointing的功能？或者是否还有什么别的省显存的方式？

1

### 请提出你的问题 Please ask your question 请问paddle中是否有类似gradient_checkpointing的功能？因为如果不开tensor和pipline并行的话似乎paddle的sharding并不会节省太多显存，或者是否还有什么别的省显存的方式？现在4卡微调7b模型batch_size开到2都会爆掉