axolotl
axolotl copied to clipboard
Guide For Multi-Node Distributed Finetuning
Title: Distributed Finetuning for Multi-Node Setup Guide
Description: This PR introduces a comprehensive guide for setting up a distributed finetuning environment using Axolotl and Accelerate. The guide covers the following steps:
- Configuring SSH for passwordless access across multiple nodes
- Generating and exchanging public keys for secure communication
- Configuring Axolotl with shared settings and host files
- Configuring Accelerate for multi-node training with Deepspeed
- Running distributed finetuning using Accelerate