OpenDelta
OpenDelta copied to clipboard
thunlp
A plug-and-play library for parameter-efficient-tuning (Delta Tuning)
Hi, I was trying to run BitFit for T5 and was getting some errors while initializing the bias parameters. It seems like in you register the bias parameter later to...
Hi @telxt, @ShengdingHu, @WuNein, thank you for creating this useful repository. I am trying to understand how to use this repo for some use cases. I see that a lot...
Hi @telxt, @ShengdingHu, @WuNein, thanks for creating the amazing repository. I was trying to perform inference on some of the results from the papers and was trying to locate the...
Hi, I saw on the documentation page there is a page for multi-task training: https://opendelta.readthedocs.io/en/latest/notes/pluginunplug.html. However I think it is not entirely clear how this modelling approach would work in...
## Bug Attempting to import `opendelta` results in an error caused by a module not found failure stemming from the [`turtle` graphics package](https://docs.python.org/3/library/turtle.html). Is it possible that the following line...
Hello @ShengdingHu, Are you able to confirm whether Flash Attention will be compatible with Open Delta LoRA? For example: ```python tokenizer = AutoTokenizer.from_pretrained("EleutherAI/pythia-1.4b") tokenizer.pad_token = tokenizer.mask_token model = GPTNeoXForCausalLM.from_pretrained("EleutherAI/pythia-1.4b") max_positions...
When performing prefix on gpt-2, it will report error: not implemented. I think that's because the pre-fix module is not recognized as GPTAttention in new_modules_like function (director: delta_modules/prefix.py).
I've been attempting to fine-tune a GPT-2 base model using Adapter from OpenDelta. While training the model, I came across this error: element 0 of tensors does not require grad...
When training MiniCPM (https://huggingface.co/openbmb/MiniCPM-2B-sft-bf16/) using OpenDelta, I ran into a type conversion error when saving the model delta_model.save_finetuned("./delta_model"): File "/opendelta/utils/model_md5.py", line 26, in gen_parameter_hash md5.update(x.cpu().numpy().data.tobytes()) TypeError: Got unsupported ScalarType BFloat16...
