NCLee

关于BGE-M3在微调时报：pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays

1

**场景**：使用BGE-M3进行finetune，数据文件.jsonl 含有158000行记录，每行记录一个query，pos列表的长度为1，neg列表的长度为15。 **异常报错**： WARNING:torch.distributed.run: ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal...

Hi teams, I'm trying to finetune something on my mac with M4 chip, the script is here: ` torchrun --nproc_per_node 1 \ -m FlagEmbedding.finetune.embedder.encoder_only.m3 \ --model_name_or_path /Users/nc/python/FlagNew/models/bge-m3 \ --train_data /Users/nc/python/FlagNew/examples/finetune/embedder/example_data/sts/sts.jsonl...