llm-foundry
llm-foundry copied to clipboard
Published
20 hours ago •
mosaicml
mosaicml
ERROR:composer.cli.launcher:Global rank 0 (PID 208865) exited with code -11
When I am doing finetune llama3.1 does the following error occurs, can't locate the exact error, how to fix it please?
Running environment:
Python 3.11.0rc1
GPU: 2xA100
CUDA Version: 12.2
Full log:
2024-08-30 11:05:27.361223: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-08-30 11:05:31,112: rank0[208865][MainThread]: DEBUG: llmfoundry.command_utils.train: Initializing dist with device...
2024-08-30 11:05:31,291: rank0[208865][MainThread]: DEBUG: llmfoundry.command_utils.train: Testing barrier with device...
2024-08-30 11:05:32,167: rank0[208865][MainThread]: DEBUG: llmfoundry.command_utils.train: Barrier test passed with device.
/Workspace/Shared/Groups/a100-shared-group/andreihuang/llm-foundry/llmfoundry/utils/config_utils.py:527: UserWarning: Setting `sync_module_states = True` for FSDP. This is required when using mixed initialization.
warnings.warn((
2024-08-30 11:05:32,217: rank0[208865][MainThread]: INFO: llmfoundry.command_utils.train: Building tokenizer...
2024-08-30 11:05:32,927: rank0[208865][MainThread]: INFO: llmfoundry.command_utils.train: Building train loader...
2024-08-30 11:05:32,929: rank0[208865][MainThread]: INFO: llmfoundry.data.finetuning.tasks: No preprocessor was supplied and no preprocessing function is registered for dataset name "json". No additional preprocessing will be applied. If the dataset is already formatted correctly, you can ignore this message.
/Workspace/Shared/Groups/a100-shared-group/andreihuang/llm-foundry/llmfoundry/data/finetuning/tasks.py:991: UserWarning: Dropped 338 examples where the prompt was longer than 4096, the prompt or response was empty, or the response was all padding tokens.
warnings.warn(
2024-08-30 11:05:42,479: rank0[208865][MainThread]: DEBUG: llmfoundry.data.finetuning.tasks: Local rank 0 finished data prep
2024-08-30 11:05:50,227: rank0[208865][MainThread]: DEBUG: llmfoundry.data.finetuning.tasks: All ranks finished data prep
2024-08-30 11:05:50,229: rank0[208865][MainThread]: INFO: llmfoundry.command_utils.train: Building eval loader...
2024-08-30 11:05:50,230: rank0[208865][MainThread]: INFO: llmfoundry.data.finetuning.tasks: No preprocessor was supplied and no preprocessing function is registered for dataset name "json". No additional preprocessing will be applied. If the dataset is already formatted correctly, you can ignore this message.
2024-08-30 11:05:59,704: rank0[208865][MainThread]: DEBUG: llmfoundry.data.finetuning.tasks: Local rank 0 finished data prep
2024-08-30 11:06:07,431: rank0[208865][MainThread]: DEBUG: llmfoundry.data.finetuning.tasks: All ranks finished data prep
2024-08-30 11:06:07,433: rank0[208865][MainThread]: INFO: llmfoundry.command_utils.train: Initializing model...
/local_disk0/.ephemeral_nfs/envs/pythonEnv-8060316e-1051-4bfa-9ae2-70ff8b1ada0d/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py:957: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
warnings.warn(
/local_disk0/.ephemeral_nfs/envs/pythonEnv-8060316e-1051-4bfa-9ae2-70ff8b1ada0d/lib/python3.11/site-packages/transformers/generation/configuration_utils.py:924: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
warnings.warn(
/local_disk0/.ephemeral_nfs/envs/pythonEnv-8060316e-1051-4bfa-9ae2-70ff8b1ada0d/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py:469: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
warnings.warn(
You are attempting to use Flash Attention 2.0 without specifying a torch dtype. This might lead to unexpected behaviour
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
Flash Attention 2.0 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in LlamaForCausalLM is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `torch_dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", torch_dtype=torch.float16)`
Flash Attention 2.0 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in LlamaModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `torch_dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", torch_dtype=torch.float16)`
Loading checkpoint shards: 100%|██████████| 4/4 [00:01<00:00, 3.72it/s]
/local_disk0/.ephemeral_nfs/envs/pythonEnv-8060316e-1051-4bfa-9ae2-70ff8b1ada0d/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py:469: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
warnings.warn(
Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00, 1.38it/s]
/local_disk0/.ephemeral_nfs/envs/pythonEnv-8060316e-1051-4bfa-9ae2-70ff8b1ada0d/lib/python3.11/site-packages/transformers/generation/configuration_utils.py:924: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
warnings.warn(
2024-08-30 11:06:12,920: rank0[208865][MainThread]: INFO: llmfoundry.command_utils.train: Building trainer...
/local_disk0/.ephemeral_nfs/envs/pythonEnv-8060316e-1051-4bfa-9ae2-70ff8b1ada0d/lib/python3.11/site-packages/composer/trainer/trainer.py:256: UserWarning: `device_train_microbatch_size='auto'` may potentially fail with unexpected CUDA errors. Auto microbatching attempts to catch CUDA Out of Memory errors and adjust the batch size, but it is possible CUDA will be put into an irrecoverable state due to PyTorch bugs, e.g. integer overflow. In this case, please manually set device_train_microbatch_size explicitly to an integer instead.
warnings.warn((
2024-08-30 11:06:12,922: rank0[208865][MainThread]: INFO: composer.utils.reproducibility: Setting seed to 17
2024-08-30 11:06:12,928: rank0[208865][MainThread]: INFO: composer.trainer.trainer: Run name: llm
2024-08-30 11:06:12,930: rank0[208865][MainThread]: INFO: composer.core.state: Automatically setting data_parallel_shard to have parallelization degree 2.
/local_disk0/.ephemeral_nfs/envs/pythonEnv-8060316e-1051-4bfa-9ae2-70ff8b1ada0d/lib/python3.11/site-packages/composer/callbacks/memory_monitor.py:137: UserWarning: The memory monitor only works on CUDA devices, but the model is on cpu.
warnings.warn(f'The memory monitor only works on CUDA devices, but the model is on {model_device.type}.')
2024-08-30 11:06:14,203: rank0[208865][MainThread]: INFO: composer.trainer.trainer: Stepping schedulers every batch. To step schedulers every epoch, set `step_schedulers_every_batch=False`.
2024-08-30 11:06:14,233: rank0[208865][MainThread]: INFO: composer.utils.reproducibility: Setting seed to 17
2024-08-30 11:06:22,784: rank0[208865][MainThread]: DEBUG: composer.utils.reproducibility: Restoring the RNG state
2024-08-30 11:06:22,785: rank0[208865][MainThread]: INFO: composer.trainer.trainer: Setting seed to 17
2024-08-30 11:06:22,785: rank0[208865][MainThread]: INFO: composer.utils.reproducibility: Setting seed to 17
2024-08-30 11:06:22,790: rank0[208865][MainThread]: INFO: llmfoundry.command_utils.train: Logging config
variables:
tokenizer_name: /local_disk0/models/Meta-Llama-3.1-8B-Instruct
global_seed: 17
max_seq_len: 4096
max_seq_len: 4096
max_duration: 3ep
eval_first: false
eval_interval: 1ep
eval_subset_num_batches: -1
global_train_batch_size: 8
run_name: null
max_split_size_mb: 512
model:
name: hf_causal_lm
init_device: mixed
pretrained_model_name_or_path: /local_disk0/models/Meta-Llama-3.1-8B-Instruct
pretrained: true
use_auth_token: false
use_flash_attention_2: true
tokenizer:
name: /local_disk0/models/Meta-Llama-3.1-8B-Instruct
kwargs:
model_max_length: 4096
train_loader:
name: finetuning
dataset:
hf_name: json
hf_kwargs:
data_dir: ./data/
split: train
max_seq_len: 4096
allow_pad_trimming: false
decoder_only_format: true
shuffle: true
drop_last: true
num_workers: 8
pin_memory: false
prefetch_factor: 2
persistent_workers: true
timeout: 0
eval_loader:
name: finetuning
dataset:
hf_name: json
hf_kwargs:
data_dir: ./data/
split: test
max_seq_len: 4096
allow_pad_trimming: false
decoder_only_format: true
shuffle: false
drop_last: true
num_workers: 8
pin_memory: false
prefetch_factor: 2
persistent_workers: true
timeout: 0
scheduler:
name: cosine_with_warmup
t_warmup: 100ba
alpha_f: 0.1
optimizer:
name: decoupled_lionw
lr: 5.0e-07
betas:
- 0.9
- 0.95
weight_decay: 0.0
algorithms:
gradient_clipping:
clipping_type: norm
clipping_threshold: 1.0
seed: 17
device_eval_batch_size: 2
device_train_microbatch_size: auto
precision: amp_bf16
fsdp_config:
sharding_strategy: FULL_SHARD
mixed_precision: PURE
activation_checkpointing: true
activation_checkpointing_reentrant: false
activation_cpu_offload: false
limit_all_gathers: true
sync_module_states: true
load_monolith_rank0_only: true
progress_bar: false
log_to_console: true
console_log_interval: 1ba
callbacks:
speed_monitor:
window_size: 10
lr_monitor: {}
memory_monitor: {}
runtime_estimator: {}
load_weights_only: true
save_interval: 1ep
save_num_checkpoints_to_keep: 3
save_folder: /Volumes/tencent-ml/default/external_volume/andreihuang/{run_name}/checkpoints
n_gpus: 2
device_train_batch_size: 4
device_train_grad_accum: auto
merge: true
n_params: 8030261248
n_active_params: 8030261248
n_trainable_params: 8030261248
2024-08-30 11:06:23,098: rank0[208865][MainThread]: INFO: llmfoundry.command_utils.train: Starting training...
2024-08-30 11:06:23,098: rank0[208865][MainThread]: INFO: composer.trainer.trainer: Using precision Precision.AMP_BF16
[2024-08-30 11:06:23,240] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
ERROR:composer.cli.launcher:Rank 1 crashed with exit code -11.
Waiting up to 30 seconds for all training processes to terminate. Press Ctrl-C to exit immediately.
Global rank 0 (PID 208865) exited with code -11
Global rank 1 (PID 208866) exited with code -11
----------Begin global rank 1 STDOUT----------
variables:
tokenizer_name: /local_disk0/models/Meta-Llama-3.1-8B-Instruct
global_seed: 17
max_seq_len: 4096
max_seq_len: 4096
max_duration: 3ep
eval_first: false
eval_interval: 1ep
eval_subset_num_batches: -1
global_train_batch_size: 8
run_name: null
max_split_size_mb: 512
model:
name: hf_causal_lm
init_device: mixed
pretrained_model_name_or_path: /local_disk0/models/Meta-Llama-3.1-8B-Instruct
pretrained: true
use_auth_token: false
use_flash_attention_2: true
tokenizer:
name: /local_disk0/models/Meta-Llama-3.1-8B-Instruct
kwargs:
model_max_length: 4096
train_loader:
name: finetuning
dataset:
hf_name: json
hf_kwargs:
data_dir: ./data/
split: train
max_seq_len: 4096
allow_pad_trimming: false
decoder_only_format: true
shuffle: true
drop_last: true
num_workers: 8
pin_memory: false
prefetch_factor: 2
persistent_workers: true
timeout: 0
eval_loader:
name: finetuning
dataset:
hf_name: json
hf_kwargs:
data_dir: ./data/
split: test
max_seq_len: 4096
allow_pad_trimming: false
decoder_only_format: true
shuffle: false
drop_last: true
num_workers: 8
pin_memory: false
prefetch_factor: 2
persistent_workers: true
timeout: 0
scheduler:
name: cosine_with_warmup
t_warmup: 100ba
alpha_f: 0.1
optimizer:
name: decoupled_lionw
lr: 5.0e-07
betas:
- 0.9
- 0.95
weight_decay: 0.0
algorithms:
gradient_clipping:
clipping_type: norm
clipping_threshold: 1.0
seed: 17
device_eval_batch_size: 2
device_train_microbatch_size: auto
precision: amp_bf16
fsdp_config:
sharding_strategy: FULL_SHARD
mixed_precision: PURE
activation_checkpointing: true
activation_checkpointing_reentrant: false
activation_cpu_offload: false
limit_all_gathers: true
sync_module_states: true
load_monolith_rank0_only: true
progress_bar: false
log_to_console: true
console_log_interval: 1ba
callbacks:
speed_monitor:
window_size: 10
lr_monitor: {}
memory_monitor: {}
runtime_estimator: {}
load_weights_only: true
save_interval: 1ep
save_num_checkpoints_to_keep: 3
save_folder: ./{run_name}/checkpoints
n_gpus: 2
device_train_batch_size: 4
device_train_grad_accum: auto
merge: true
n_params: 8030261248
n_active_params: 8030261248
n_trainable_params: 8030261248
[2024-08-30 11:06:23,240] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
----------End global rank 1 STDOUT----------
----------Begin global rank 1 STDERR----------
2024-08-30 11:05:27.438049: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-08-30 11:05:31,077: rank1[208866][MainThread]: DEBUG: llmfoundry.command_utils.train: Initializing dist with device...
2024-08-30 11:05:31,287: rank1[208866][MainThread]: DEBUG: llmfoundry.command_utils.train: Testing barrier with device...
2024-08-30 11:05:32,166: rank1[208866][MainThread]: DEBUG: llmfoundry.command_utils.train: Barrier test passed with device.
/Workspace/Shared/Groups/a100-shared-group/andreihuang/llm-foundry/llmfoundry/utils/config_utils.py:527: UserWarning: Setting `sync_module_states = True` for FSDP. This is required when using mixed initialization.
warnings.warn((
2024-08-30 11:05:32,217: rank1[208866][MainThread]: INFO: llmfoundry.command_utils.train: Building tokenizer...
2024-08-30 11:05:32,927: rank1[208866][MainThread]: INFO: llmfoundry.command_utils.train: Building train loader...
2024-08-30 11:05:32,929: rank1[208866][MainThread]: INFO: llmfoundry.data.finetuning.tasks: No preprocessor was supplied and no preprocessing function is registered for dataset name "json". No additional preprocessing will be applied. If the dataset is already formatted correctly, you can ignore this message.
2024-08-30 11:05:32,929: rank1[208866][MainThread]: DEBUG: llmfoundry.data.finetuning.tasks: Waiting for local_rank 0 to finish data prep
/Workspace/Shared/Groups/a100-shared-group/andreihuang/llm-foundry/llmfoundry/data/finetuning/tasks.py:991: UserWarning: Dropped 338 examples where the prompt was longer than 4096, the prompt or response was empty, or the response was all padding tokens.
warnings.warn(
2024-08-30 11:05:50,227: rank1[208866][MainThread]: DEBUG: llmfoundry.data.finetuning.tasks: All ranks finished data prep
2024-08-30 11:05:50,228: rank1[208866][MainThread]: INFO: llmfoundry.command_utils.train: Building eval loader...
2024-08-30 11:05:50,229: rank1[208866][MainThread]: INFO: llmfoundry.data.finetuning.tasks: No preprocessor was supplied and no preprocessing function is registered for dataset name "json". No additional preprocessing will be applied. If the dataset is already formatted correctly, you can ignore this message.
2024-08-30 11:05:50,229: rank1[208866][MainThread]: DEBUG: llmfoundry.data.finetuning.tasks: Waiting for local_rank 0 to finish data prep
2024-08-30 11:06:07,430: rank1[208866][MainThread]: DEBUG: llmfoundry.data.finetuning.tasks: All ranks finished data prep
2024-08-30 11:06:07,431: rank1[208866][MainThread]: INFO: llmfoundry.command_utils.train: Initializing model...
/local_disk0/.ephemeral_nfs/envs/pythonEnv-8060316e-1051-4bfa-9ae2-70ff8b1ada0d/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py:957: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
warnings.warn(
You are attempting to use Flash Attention 2.0 without specifying a torch dtype. This might lead to unexpected behaviour
Flash Attention 2.0 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in LlamaForCausalLM is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `torch_dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", torch_dtype=torch.float16)`
Flash Attention 2.0 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in LlamaModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `torch_dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", torch_dtype=torch.float16)`
/local_disk0/.ephemeral_nfs/envs/pythonEnv-8060316e-1051-4bfa-9ae2-70ff8b1ada0d/lib/python3.11/site-packages/transformers/generation/configuration_utils.py:924: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
warnings.warn(
2024-08-30 11:06:12,911: rank1[208866][MainThread]: INFO: llmfoundry.command_utils.train: Building trainer...
/local_disk0/.ephemeral_nfs/envs/pythonEnv-8060316e-1051-4bfa-9ae2-70ff8b1ada0d/lib/python3.11/site-packages/composer/trainer/trainer.py:256: UserWarning: `device_train_microbatch_size='auto'` may potentially fail with unexpected CUDA errors. Auto microbatching attempts to catch CUDA Out of Memory errors and adjust the batch size, but it is possible CUDA will be put into an irrecoverable state due to PyTorch bugs, e.g. integer overflow. In this case, please manually set device_train_microbatch_size explicitly to an integer instead.
warnings.warn((
2024-08-30 11:06:12,922: rank1[208866][MainThread]: INFO: composer.utils.reproducibility: Setting seed to 18
2024-08-30 11:06:12,928: rank1[208866][MainThread]: INFO: composer.trainer.trainer: Run name: llm
2024-08-30 11:06:12,930: rank1[208866][MainThread]: INFO: composer.core.state: Automatically setting data_parallel_shard to have parallelization degree 2.
2024-08-30 11:06:14,219: rank1[208866][MainThread]: INFO: composer.trainer.trainer: Stepping schedulers every batch. To step schedulers every epoch, set `step_schedulers_every_batch=False`.
2024-08-30 11:06:14,233: rank1[208866][MainThread]: INFO: composer.utils.reproducibility: Setting seed to 17
2024-08-30 11:06:21,766: rank1[208866][MainThread]: DEBUG: composer.utils.reproducibility: Restoring the RNG state
2024-08-30 11:06:21,767: rank1[208866][MainThread]: INFO: composer.trainer.trainer: Setting seed to 18
2024-08-30 11:06:21,767: rank1[208866][MainThread]: INFO: composer.utils.reproducibility: Setting seed to 18
2024-08-30 11:06:21,773: rank1[208866][MainThread]: INFO: llmfoundry.command_utils.train: Logging config
2024-08-30 11:06:23,198: rank1[208866][MainThread]: INFO: llmfoundry.command_utils.train: Starting training...
2024-08-30 11:06:23,198: rank1[208866][MainThread]: INFO: composer.trainer.trainer: Using precision Precision.AMP_BF16
ERROR:composer.cli.launcher:Global rank 0 (PID 208865) exited with code -11
----------End global rank 1 STDERR----------
