Einengutenmorgen

Results 7 comments of Einengutenmorgen

ERROR:composer.cli.launcher:Rank 2 crashed with exit code -7

I have the same problem but im using a ml.g4dn.12xlarge (4x Tesla T4). I'm running it on AWS Sagemaker inside a docker (recommended docker image) and im using the mpt-7b_dolly_sft.yaml

Fused Cross Entropy is not installed. Either (1) have a CUDA-compatible GPU and `pip install .[gpu]`, or (2) set your config model.loss_fn=torch_crossentropy.

Im having the same issue. Im using the docker image [2.0.1_cu117-python3.10-ubuntu20.04-aws] and don't use a Venv. I can run the hf_ generate.py but I can't run train.py

Fused Cross Entropy is not installed. Either (1) have a CUDA-compatible GPU and `pip install .[gpu]`, or (2) set your config model.loss_fn=torch_crossentropy.

thanks my mistake

Fine-Tuning a model with SageMaker

What's your Question ?

TypeError: BFloat16 is not supported on MPS

Probably: BFloat16 is not supported on MPS

Composer container image doesn't show exceptions in logs when Trainer crashes

Have the same issue. Output looks exactly like mine. But I don't get an OOM like bandish-shah suggests. I set 'export PYTHONUNBUFFERED=True' and don't receive any new information for this...

[Feature Request]: Hybrid Search with BM25

+1