21Timothy

Has anyone ever encountered this problem before?

> `split_dataset_by_node` doesn't give the exact same number of examples to each node in the case of iterable datasets, though it tries to be as equal as possible. In particular...

设置device_map='auto'，添加了 if torch.cuda.device_count() > 1: model.hf_device_map['transformer.output_layer'] = model.hf_device_map['transformer.embedding'] new_hf_device_map = model.hf_device_map model.cpu() del model torch.cuda.empty_cache() print(f'memory_allocated {torch.cuda.memory_allocated()}') print('loading real model...') model = AutoModel.from_pretrained("THUDM/chatglm2-6b", trust_remote_code=True, device_map=new_hf_device_map) print(model.hf_device_map) 这段代码后依然报了那个错，把device_map='auto'去掉才运行成功的··· 但运行成功也很慢，我用的是4张p40，AdvertiseGen数据集竟然显示要1700多个小时！请问是参数问题还是就这么慢呀···@beyondguo

> Great ! Maybe it can be worth having an example that we can include in the docs for other people, did you need anything else than the Join context...