Rangehow

请教一下，在合并Chinese-Alpaca-Plus模型的时候，为什么lora模型参数不能颠倒顺序

1

在项目离线合成lora中有如下描述 > 合并Chinese-Alpaca-Plus需要提供两个LoRA权重，分别为Chinese-LLaMA-Plus-LoRA和Chinese-Alpaca-Plus-LoRA。执行以下命令完成合并： > > python scripts/merge_llama_with_chinese_lora.py \ > --base_model path_to_original_llama_hf_dir \ > --lora_model path_to_chinese_llama_plus_lora,path_to_chinese_alpaca_plus_lora \ > --output_type [pth|huggingface] \ > --output_dir path_to_output_dir > 参数选项含义与单LoRA权重合并中的含义相同。 需要注意的是 --lora_model参数后要提供两个lora_model的地址，用逗号分隔。 > > 两个LoRA模型的顺序很重要，不能颠倒。先写LLaMA-Plus-LoRA然后写Alpaca-Plus-LoRA。...

### System Info ```shell - optimum : 1.14.1 - huggingface_hub version: 0.19.4 - Platform: Linux-3.10.0-1160.88.1.el7.x86_64-x86_64-with-glibc2.17 - Python version: 3.9.17 - Running in iPython ?: No - Running in notebook ?:...

How to convert tokenizers.tokenizer to XXTokenizerFast in transformers?

3

### Motivation I followed the guide [build-a-tokenizer-from-scratch](https://huggingface.co/docs/tokenizers/quicktour#build-a-tokenizer-from-scratch) and got a single tokenizer.json from my corpus. Since I'm not sure if it is compatible with the trainer, I want to convert...

How can we resume training from lora model?

4

### System Info - `transformers` version: 4.39.0.dev0 - Platform: Linux-3.10.0-1160.71.1.el7.x86_64-x86_64-with-glibc2.17 - Python version: 3.10.13 - Huggingface_hub version: 0.21.4 - Safetensors version: 0.4.2 - Accelerate version: 0.27.2 - Accelerate config: -...

Question about RoPE code

2

I found here exist a difference in rope implementation mostly on permutation. Does this difference not affect the final result ？ I'm not quite sure what I'm thinking. Sincerely ask...

Do unsloth work on non lora case?

1

I have tried unsloth+qlora, it's cool and brings considerable speedup and vram reduction. But I am not sure if this repo is useful for **full fine-tune** after searching repo/website/benchmark. So...

不完全的提取

2

https://www.neu.edu.cn/xygk/lrld.htm 从学校领导的提取只能提取到宁恩承

我在evaluation的issue里发现了很多和hf-mirror相关的问题 ``` File "/modules/evaluate_modules/metrics/evaluate-metric--exact_match/009c8b5313309ea5b135d526433d5ee76508ba1554cbe88310a30f85bb57ec88/exact_match.py", line 16 } ``` 这个问题是因为不完全下载导致的吗？

这个训练上比FA2更快，而且vram占用更少，在llama2的测试上非常有效。然而还不支持原生的qwen2，尽管有些三方脚本支持llamafy qwen，但是因为潜在的实现错误风险，让人不太有尝试欲望。unsloth已经被集成到llamafactory中。 benchmark：https://unsloth.ai/blog/mistral-benchmark#Benchmark%20tables 仓库：https://github.com/unslothai/unsloth

accelerate Inappropriate convert all inputs to specific dtype

5

As title said, I made a minium snippet like this: ```python from transformers import ,Trainer class MyTrainer(Trainer): def compute_loss(inputs): input_ids = inputs.pop("input_ids") attention_mask = inputs.pop("attention_mask") valid_label_index_list = inputs.pop( "valid_label_index_list" )...