Ayush Bihani

This is the accelerate config I am using ```yaml compute_environment: LOCAL_MACHINE distributed_type: FSDP downcast_bf16: 'no' fsdp_config: fsdp_auto_wrap_policy: NO_WRAP fsdp_backward_prefetch_policy: BACKWARD_PRE fsdp_offload_params: false fsdp_sharding_strategy: 1 fsdp_state_dict_type: FULL_STATE_DICT machine_rank: 0 main_training_function: main...

Yes I tried out with `run_clm_no_trainer.py` as well. There the code block was as below: ```bash accelerate launch run_clm_no_trainer.py \ --dataset_name wikitext \ --dataset_config_name wikitext-2-raw-v1 \ --model_name_or_path gpt2 \ --output_dir...

> This is the accelerate config I am using > > ```yaml > compute_environment: LOCAL_MACHINE > distributed_type: FSDP > downcast_bf16: 'no' > fsdp_config: > fsdp_auto_wrap_policy: NO_WRAP > fsdp_backward_prefetch_policy: BACKWARD_PRE >...

In this I could not find reference to what should be the auto wrap policy. Among the options should I go with `GPTJBlock`?

Hi I am getting this error: I used the `GPTJBlock` ```bash Traceback (most recent call last): File "run_clm_no_trainer.py", line 685, in main() File "run_clm_no_trainer.py", line 510, in main model, optimizer,...

> Yes, also, next time, please use the forums for suggestions or help as this isn't an issue per se. And make sure to provide the entire info required while...

> I thought you were using GPTJ but the command above is using `gpt2 `. Please refer the modeling code in transformers for getting the transformer block name. For GPT2,...

Hi I modified that and I got `RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling 'cublasCreate(handle)'`

@pacman100 Updated accelerate env: ``` Copy-and-paste the text below in your GitHub issue - `Accelerate` version: 0.18.0 - Platform: Linux-5.4.0-136-generic-x86_64-with-glibc2.10 - Python version: 3.8.12 - Numpy version: 1.22.2 - PyTorch...

> @pacman100 Updated accelerate env: > > ``` > Copy-and-paste the text below in your GitHub issue > > - `Accelerate` version: 0.18.0 > - Platform: Linux-5.4.0-136-generic-x86_64-with-glibc2.10 > - Python...