Why is S3 file system being used?

[phalexo]

/arrow/cpp/src/arrow/filesystem/s3fs.cc:2598: arrow::fs::FinalizeS3 was not called even though S3 was initialized. This could lead to a segmentation fault at exit Segmentation fault (core dumped)

Is it possible to switch it off or suppress its use?

Or alternatively to fix this issue?

[RickyWang111]

have you solve it ? I have the same problem.

[jonataslaw]

Just comment out:

https://github.com/TimDettmers/bitsandbytes/blob/1b8772a8f33fdb47df0c849302cbb7e703571b8c/bitsandbytes/functional.py#L187

[kukrishna]

I also got the same error. @jonataslaw's fix works for me. Although I am not sure what impact it would have on speed of the code.

[kukrishna]

The error also goes away for me if I use adamw_8bit instead of paged_adamw_8bit as the optimizer

[AcidChristLab]

A simple solution to fix this error:

pip install --upgrade --force-reinstall pyarrow==11.0.0

[frederictost]

The error also goes away for me if I use adamw_8bit instead of paged_adamw_8bit as the optimizer

It has solved the issue (Error operation not supported at line 380 in file /mmfs1/gscratch/zlab/timdettmers/git/bitsandbytes/csrc/pythonInterface.c) I had when adding target_modules=["query_key_value", "dense", "dense_h_to_4h", "dense_4h_to_h", ],

[theobjectivedad]

Downgrading to pyarrow 11 worked for me and allows me to use paged_adamw_32bit. The issue happens whenever I resume training. I don't see a downside to the downgrade however it may make sense to add something like pyarrow<12 to requirements.

[Tunneller]

I was getting this error whenever I tried to ^C from a program, and it would then freeze my terminal window. Frustrating. Downgrading to 11.0.0 fixed the problem