Tesla T4 env to finetuning for err msg: error: ‘for’ loop initial declarations are only allowed in C99 mode

[JiesenSun]

/tmp/tmpp86myte5/main.c: In function ‘list_to_cuuint64_array’: /tmp/tmpp86myte5/main.c:354:3: error: ‘for’ loop initial declarations are only allowed in C99 mode for (Py_ssize_t i = 0; i < len; i++) { ^ /tmp/tmpp86myte5/main.c:354:3: note: use option -std=c99 or -std=gnu99 to compile your code /tmp/tmpp86myte5/main.c: In function ‘list_to_cuuint32_array’: /tmp/tmpp86myte5/main.c:365:3: error: ‘for’ loop initial declarations are only allowed in C99 mode for (Py_ssize_t i = 0; i < len; i++) { ^ Traceback (most recent call last): File "/opt/prj/notebook/sft_test.py", line 121, in trainer_stats = trainer.train() ^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/trl/trainer/sft_trainer.py", line 361, in train output = super().train(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/transformers/trainer.py", line 1859, in train return inner_training_loop( ^^^^^^^^^^^^^^^^^^^^ File "", line 361, in _fast_inner_training_loop File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/transformers/trainer.py", line 3138, in training_step loss = self.compute_loss(model, inputs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/transformers/trainer.py", line 3161, in compute_loss outputs = model(**inputs) ^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/accelerate/utils/operations.py", line 825, in forward return model_forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/accelerate/utils/operations.py", line 813, in call return convert_to_fp32(self.model_forward(*args, **kwargs)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/torch/amp/autocast_mode.py", line 16, in decorate_autocast return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/unsloth/models/llama.py", line 882, in PeftModelForCausalLM_fast_forward return self.base_model( ^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/peft/tuners/tuners_utils.py", line 161, in forward return self.model.forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/accelerate/hooks.py", line 166, in new_forward output = module._old_forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/unsloth/models/llama.py", line 813, in _CausalLM_fast_forward outputs = self.model( ^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/accelerate/hooks.py", line 166, in new_forward output = module._old_forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/unsloth/models/llama.py", line 650, in LlamaModel_fast_forward hidden_states = Unsloth_Offloaded_Gradient_Checkpointer.apply( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/torch/autograd/function.py", line 553, in apply return super().apply(*args, **kwargs) # type: ignore[misc] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/torch/cuda/amp/autocast_mode.py", line 115, in decorate_fwd return fwd(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/unsloth/models/_utils.py", line 333, in forward (output,) = forward_function(hidden_states, *args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/accelerate/hooks.py", line 166, in new_forward output = module._old_forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/unsloth/models/llama.py", line 432, in LlamaDecoderLayer_fast_forward hidden_states = fast_rms_layernorm(self.input_layernorm, hidden_states) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/unsloth/kernels/rms_layernorm.py", line 190, in fast_rms_layernorm out = Fast_RMS_Layernorm.apply(X, W, eps, gemma) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/torch/autograd/function.py", line 553, in apply return super().apply(*args, **kwargs) # type: ignore[misc] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/unsloth/kernels/rms_layernorm.py", line 144, in forward fx[(n_rows,)]( File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/triton/runtime/jit.py", line 550, in run bin.c_wrapper( ^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/triton/compiler/compiler.py", line 692, in getattribute self._init_handles() File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/triton/compiler/compiler.py", line 670, in _init_handles bin_path = {driver.HIP: "hsaco_path", driver.CUDA: "cubin"}[driver.backend] ^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/triton/runtime/driver.py", line 157, in getattr self._initialize_obj() File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/triton/runtime/driver.py", line 154, in _initialize_obj self._obj = self._init_fn() ^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/triton/runtime/driver.py", line 187, in initialize_driver return CudaDriver() ^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/triton/runtime/driver.py", line 77, in init self.utils = CudaUtils() ^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/triton/runtime/driver.py", line 47, in init so = _build("cuda_utils", src_path, tmpdir) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/triton/common/build.py", line 106, in _build ret = subprocess.check_call(cc_cmd) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/sft_env/lib/python3.11/subprocess.py", line 413, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/usr/bin/gcc', '/tmp/tmpp86myte5/main.c', '-O3', '-I/opt/anaconda3/envs/sft_env/lib/python3.11/site-packages/triton/common/../third_party/cuda/include', '-I/opt/anaconda3/envs/sft_env/include/python3.11', '-I/tmp/tmpp86myte5', '-shared', '-fPIC', '-lcuda', '-o', '/tmp/tmpp86myte5/cuda_utils.cpython-311-x86_64-linux-gnu.so', '-L/lib64', '-L/lib', '-L/lib64', '-L/lib']' returned non-zero exit status 1. 0%| | 0/60 [00:03<?, ?it/s]

[danielhanchen]

@JiesenSun Oh my that is a super weird error msg - sadly I can't be of much help - I would try reinstalling :(

[JiesenSun]

@JiesenSun Oh my that is a super weird error msg - sadly I can't be of much help - I would try reinstalling :(

3Q for your reply。I find this reason，because gcc version is too low，upgrade and solved。

[danielhanchen]

Oh great you solved it! Sorry I wasn't helpful!

[Sunny5062]

my gcc version is 14.1.0 and still get the same error

[Sunny5062]

nevermind i figured it out

[danielhanchen]

Oh ok! @Sunny5062 Could you elaborate how you solved it? :)

[Sunny5062]

I downloaded mingw from winlibs which gave me errors. Then I switched to downloading from sourceforges, which did not cause any errors.