Any plan to support PEFT LoRA models?

[zsc]

I played with the hf_generate branch and it seems quite ready to be expanded to support BLOOM-3B/7B1 models etc (https://github.com/zsc/tau/pull/1 ).

Great work!

Is there an imminent plan to support PEFT LoRA models? They are not that different, but HFTracer is not happy with the extra layer of LoraModel https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora.py .

Traceback (most recent call last):
  File "/home/zsc/.local/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
    fn(i, *args)
  File "/data/tau/pippy/utils.py", line 282, in run_worker
    run_func(my_pp_ranks, args, *extra_args)
  File "/data/tau/examples/inference/opt_generate.py", line 74, in run_all
    pipe_driver, stage_mod = pippy.all_compile(
  File "/data/tau/pippy/compile.py", line 176, in all_compile
    return _compile(
  File "/data/tau/pippy/compile.py", line 78, in _compile
    pipe_model = Pipe.from_tracing(
  File "/data/tau/pippy/IR.py", line 1010, in from_tracing
    graph = _pipeline_tracer.trace(mod, **kwargs)
  File "/data/tau/pippy/hf/utils.py", line 267, in trace
    graph = super().trace(*args, **kwargs)
  File "/data/transformers/src/transformers/utils/fx.py", line 1017, in trace
    raise RuntimeError(
RuntimeError: Could not generate input named input_ids for because root is not a 
transformers.PreTrainedModel.

With the support LoRA model, there will be a way to fine-tune large models like LLaMA-65B models in https://github.com/tloen/alpaca-lora/ , but at much faster speed thanks to the true pipeline parallelism bestowed by Tau.

[kwen2501]

Hi @zsc , first off, thanks much for the PR in https://github.com/zsc/tau/pull/1. Would you mind pulling it against the upstream repo? We will make sure to include it in our example refactoring.

For the PEFT support, let me reach out to our Hugging Face contact regarding HFTracer and see if there is an easy path. Thanks for suggesting a new use 👍

[jph00]

@kwen2501 just checking whether you had any news regarding peft/lora support?