FedML icon indicating copy to clipboard operation
FedML copied to clipboard

Weights&biases

Open Yang-cpu502 opened this issue 2 years ago • 2 comments

When I look at the result display graph in Weights&biases, it shows that the operation failed, check the reason, is it because of insufficient memory?

wandb: Run data is saved locally in /home/hadoop/下载/FedML-master/fedml_experiments/standalone/fedavg/wandb/run-20220408_112803-2ots05l0 wandb: Run wandb offline to turn off syncing. wandb: Syncing run FedAVG-r200-e1-lr0.03 wandb: ⭐️ View project at https://wandb.ai/yangbinbing/fedml wandb: 🚀 View run at https://wandb.ai/yangbinbing/fedml/runs/2ots05l0 INFO:root:load_data. dataset_name = mnist Traceback (most recent call last): File "./main_fedavg.py", line 305, in dataset = load_data(args, args.dataset) File "./main_fedavg.py", line 110, in load_data class_num = load_partition_data_mnist(args.batch_size) File "/home/hadoop/下载/FedML-master/fedml_api/data_preprocessing/MNIST/data_loader.py", line 90, in load_partition_data_mnist users, groups, train_data, test_data = read_data(train_path, test_path) File "/home/hadoop/下载/FedML-master/fedml_api/data_preprocessing/MNIST/data_loader.py", line 33, in read_data cdata = json.load(inf) File "/home/hadoop/miniconda/envs/fedml/lib/python3.7/json/init.py", line 293, in load return loads(fp.read(), File "/home/hadoop/miniconda/envs/fedml/lib/python3.7/codecs.py", line 322, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) MemoryError

wandb: Waiting for W&B process to finish... (failed 1). Press Control-C to abort syncing. wandb:
wandb: Synced FedAVG-r200-e1-lr0.03: https://wandb.ai/yangbinbing/fedml/runs/2ots05l0 wandb: Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) wandb: Find logs at: ./wandb/run-20220408_112803-2ots05l0/logs

Yang-cpu502 avatar Apr 08 '22 03:04 Yang-cpu502

@Yang-cpu502 Please check your memory cost when running.

chaoyanghe avatar Apr 15 '22 19:04 chaoyanghe

@Yang-cpu502 is your issue solved?

chaoyanghe avatar Aug 19 '22 17:08 chaoyanghe

Closing due to inactivity.

fedml-dimitris avatar Oct 24 '23 19:10 fedml-dimitris