Falcon LoRA + LoRA config changes + DS sampling inference

[jordiclive]

• Add DeepSpeed for the sampling report. Needed to run generation with 65B sft, 65B LoRA, 40B sft. (WiP, hacked a new script which works).
• Add LoRA, for Falcon
• Add loading function of a fine-tuned LoRA to circumvent a bug with DS offload inflight parameters when loading fine-tuned LoRAs with the PEFT library.
• Change default Lora configs to what is suggested by QLoRA: They recommend:

1. r = 64 (although can be much lower, e.g. 16), α = 16 (proportional to lr so not critical)
2. Add LoRA modules on all linear layers of the model.
3. Adam beta2 = 0.999,
4. Max grad norm of 0.3
5. LoRA dropout of 0.1 for models up to 13B and 0.05 for 33B and 65B models

[github-actions[bot]]

:x: pre-commit failed. Please run pre-commit run --all-files locally and commit the changes. Find more information in the repository's CONTRIBUTING.md

[github-actions[bot]]

:x: pre-commit failed. Please run pre-commit run --all-files locally and commit the changes. Find more information in the repository's CONTRIBUTING.md

[github-actions[bot]]

:x: pre-commit failed. Please run pre-commit run --all-files locally and commit the changes. Find more information in the repository's CONTRIBUTING.md

[github-actions[bot]]

:x: pre-commit failed. Please run pre-commit run --all-files locally and commit the changes. Find more information in the repository's CONTRIBUTING.md

[github-actions[bot]]

:x: pre-commit failed. Please run pre-commit run --all-files locally and commit the changes. Find more information in the repository's CONTRIBUTING.md

[github-actions[bot]]

:x: pre-commit failed. Please run pre-commit run --all-files locally and commit the changes. Find more information in the repository's CONTRIBUTING.md