[QUESTION] Checkpoint storage format

[syx11237744]

Your question Ask a clear and concise question about Megatron-LM. Could you let me know which version I should revert to if I want to use the previous checkpoint storage format, which is stored as .pt? Or are there any other methods to save it as a .pt file?Thank you!

DATASET_PATH=/share/root/out_file/sum.jsonl SAVE_PATH=/share/sunyuanxu/out_file/sum VOCAB_FILE=gpt2_/vocab.json MERGE_FILE=gpt2_/merges.txt

export CUDA_DEVICE_MAX_CONNECTIONS=1

GPUS_PER_NODE=1

Change for multinode config

MASTER_ADDR=localhost MASTER_PORT=6000 NNODES=1 NODE_RANK=0 WORLD_SIZE=$(($GPUS_PER_NODE*$NNODES))

CHECKPOINT_PATH=/share/root/checkpoint/cp DATA_PATH=/share/root/out_file/sum_text_document DISTRIBUTED_ARGS=" --nproc_per_node $GPUS_PER_NODE
--nnodes $NNODES
--node_rank $NODE_RANK
--master_addr $MASTER_ADDR
--master_port $MASTER_PORT "

GPT_ARGS=" --num-layers 24
--hidden-size 1024
--num-attention-heads 16
--seq-length 1024
--max-position-embeddings 1024
--micro-batch-size 32
--global-batch-size 256
--lr 0.00015
--train-iters 1000
--lr-decay-iters 320000
--lr-decay-style cosine
--min-lr 1.0e-5
--weight-decay 1e-2
--lr-warmup-fraction .01
--clip-grad 1.0
--fp16
--attention-softmax-in-fp32
"

DATA_ARGS=" --data-path $DATA_PATH
--vocab-file $VOCAB_FILE
--merge-file $MERGE_FILE
--split 949,50,1 "

OUTPUT_ARGS=" --log-interval 100
--save-interval 10000
--eval-interval 1000
--eval-iters 10 "

torchrun $DISTRIBUTED_ARGS Megatron-LM/pretrain_gpt.py
$GPT_ARGS
$DATA_ARGS
$OUTPUT_ARGS
--distributed-backend nccl
--save $CHECKPOINT_PATH
--load $CHECKPOINT_PATH