axolotl icon indicating copy to clipboard operation
axolotl copied to clipboard

Guide For Multi-Node Distributed Finetuning

Open shahdivax opened this issue 10 months ago • 2 comments

Title: Distributed Finetuning for Multi-Node Setup Guide

Description: This PR introduces a comprehensive guide for setting up a distributed finetuning environment using Axolotl and Accelerate. The guide covers the following steps:

  1. Configuring SSH for passwordless access across multiple nodes
  2. Generating and exchanging public keys for secure communication
  3. Configuring Axolotl with shared settings and host files
  4. Configuring Accelerate for multi-node training with Deepspeed
  5. Running distributed finetuning using Accelerate

shahdivax avatar Apr 04 '24 07:04 shahdivax