ray_lightning
ray_lightning copied to clipboard
Published
20 hours ago •
ray-project
ray-project
`ray_ddp` the progressive bar is broken
The progressive bar in the ray_ddp shows the results:
Epoch 0: 3%|▎ | 32/937 [00:00<00:06, 131.01it/s, loss=1.83, v_num=2]
Epoch 0: 3%|▎ | 32/937 [00:00<00:06, 131.70it/s, loss=1.83, v_num=2]
Epoch 0: 5%|▌ | 50/937 [00:00<00:06, 143.88it/s, loss=1.67, v_num=2]
Epoch 0: 5%|▌ | 50/937 [00:00<00:06, 144.36it/s, loss=1.67, v_num=2]
Epoch 0: 7%|▋ | 67/937 [00:00<00:05, 149.46it/s, loss=1.72, v_num=2]
Epoch 0: 7%|▋ | 68/937 [00:00<00:05, 150.15it/s, loss=1.74, v_num=2]
Epoch 0: 7%|▋ | 68/937 [00:00<00:05, 149.76it/s, loss=1.74, v_num=2]
Epoch 0: 9%|▉ | 86/937 [00:00<00:05, 154.51it/s, loss=1.6, v_num=2]
Epoch 0: 9%|▉ | 86/937 [00:00<00:05, 154.74it/s, loss=1.6, v_num=2]
Epoch 0: 9%|▉ | 87/937 [00:00<00:05, 154.71it/s, loss=1.62, v_num=2]
Epoch 0: 9%|▉ | 87/937 [00:00<00:05, 154.94it/s, loss=1.62, v_num=2]
Epoch 0: 11%|█ | 104/937 [00:00<00:05, 156.98it/s, loss=1.58, v_num=2]
Epoch 0: 11%|█ | 105/937 [00:00<00:05, 157.47it/s, loss=1.59, v_num=2]
Epoch 0: 11%|█ | 105/937 [00:00<00:05, 157.07it/s, loss=1.59, v_num=2]
Epoch 0: 13%|█▎ | 123/937 [00:00<00:05, 159.10it/s, loss=1.58, v_num=2]
Epoch 0: 13%|█▎ | 123/937 [00:00<00:05, 159.33it/s, loss=1.58, v_num=2]
Epoch 0: 15%|█▌ | 141/937 [00:00<00:04, 160.96it/s, loss=1.6, v_num=2]
Epoch 0: 15%|█▌ | 141/937 [00:00<00:04, 161.13it/s, loss=1.6, v_num=2]
Epoch 0: 17%|█▋ | 159/937 [00:00<00:04, 162.49it/s, loss=1.55, v_num=2]
Epoch 0: 17%|█▋ | 159/937 [00:00<00:04, 162.66it/s, loss=1.55, v_num=2]
Epoch 0: 17%|█▋ | 160/937 [00:00<00:04, 162.58it/s, loss=1.55, v_num=2]
Epoch 0: 17%|█▋ | 160/937 [00:00<00:04, 162.75it/s, loss=1.59, v_num=2]
Epoch 0: 17%|█▋ | 160/937 [00:00<00:04, 162.53it/s, loss=1.59, v_num=2]
Epoch 0: 17%|█▋ | 161/937 [00:00<00:04, 162.84it/s, loss=1.59, v_num=2]
Epoch 0: 17%|█▋ | 161/937 [00:00<00:04, 162.58it/s, loss=1.55, v_num=2]
Epoch 0: 17%|█▋ | 161/937 [00:00<00:04, 162.78it/s, loss=1.55, v_num=2]
Epoch 0: 19%|█▉ | 179/937 [00:01<00:04, 163.51it/s, loss=1.59, v_num=2]
Epoch 0: 19%|█▉ | 179/937 [00:01<00:04, 163.66it/s, loss=1.59, v_num=2]
Epoch 0: 21%|██ | 197/937 [00:01<00:04, 164.43it/s, loss=1.55, v_num=2]
Epoch 0: 21%|██ | 197/937 [00:01<00:04, 164.47it/s, loss=1.55, v_num=2]
Epoch 0: 23%|██▎ | 215/937 [00:01<00:04, 164.94it/s, loss=1.56, v_num=2]
Epoch 0: 23%|██▎ | 215/937 [00:01<00:04, 165.06it/s, loss=1.56, v_num=2]
Epoch 0: 23%|██▎ | 215/937 [00:01<00:04, 164.91it/s, loss=1.56, v_num=2]
Epoch 0: 25%|██▍ | 233/937 [00:01<00:04, 165.57it/s, loss=1.7, v_num=2]
Epoch 0: 25%|██▍ | 233/937 [00:01<00:04, 165.72it/s, loss=1.7, v_num=2]
Epoch 0: 25%|██▍ | 234/937 [00:01<00:04, 165.75it/s, loss=1.68, v_num=2]
Epoch 0: 25%|██▍ | 234/937 [00:01<00:04, 165.60it/s, loss=1.68, v_num=2]
Epoch 0: 27%|██▋ | 252/937 [00:01<00:04, 166.08it/s, loss=1.59, v_num=2]
Epoch 0: 27%|██▋ | 252/937 [00:01<00:04, 166.19it/s, loss=1.59, v_num=2]
Epoch 0: 29%|██▉ | 270/937 [00:01<00:04, 166.58it/s, loss=1.49, v_num=2]
Epoch 0: 29%|██▉ | 270/937 [00:01<00:04, 166.70it/s, loss=1.49, v_num=2]
Epoch 0: 31%|███ | 288/937 [00:01<00:03, 166.93it/s, loss=1.52, v_num=2]
Epoch 0: 31%|███ | 288/937 [00:01<00:03, 167.09it/s, loss=1.52, v_num=2]
Epoch 0: 31%|███ | 289/937 [00:01<00:03, 167.14it/s, loss=1.52, v_num=2]
Epoch 0: 31%|███ | 289/937 [00:01<00:03, 166.92it/s, loss=1.5, v_num=2]
Epoch 0: 31%|███ | 289/937 [00:01<00:03, 167.08it/s, loss=1.5, v_num=2]
Epoch 0: 33%|███▎ | 306/937 [00:01<00:03, 166.78it/s, loss=1.57, v_num=2]
Epoch 0: 33%|███▎ | 306/937 [00:01<00:03, 166.93it/s, loss=1.57, v_num=2]
Epoch 0: 34%|███▍ | 323/937 [00:01<00:03, 166.96it/s, loss=1.41, v_num=2]
Epoch 0: 35%|███▍ | 324/937 [00:01<00:03, 167.10it/s, loss=1.42, v_num=2]
Epoch 0: 35%|███▍ | 324/937 [00:01<00:03, 166.98it/s, loss=1.42, v_num=2]
Epoch 0: 36%|███▋ | 342/937 [00:02<00:03, 167.28it/s, loss=1.37, v_num=2]
Epoch 0: 36%|███▋ | 342/937 [00:02<00:03, 167.37it/s, loss=1.37, v_num=2]
Epoch 0: 38%|███▊ | 360/937 [00:02<00:03, 167.53it/s, loss=1.42, v_num=2]
Epoch 0: 38%|███▊ | 360/937 [00:02<00:03, 167.53it/s, loss=1.42, v_num=2]
Epoch 0: 40%|████ | 378/937 [00:02<00:03, 167.80it/s, loss=1.37, v_num=2]
Epoch 0: 40%|████ | 378/937 [00:02<00:03, 167.84it/s, loss=1.37, v_num=2]
Epoch 0: 42%|████▏ | 395/937 [00:02<00:03, 167.88it/s, loss=1.34, v_num=2]
Epoch 0: 42%|████▏ | 396/937 [00:02<00:03, 168.00it/s, loss=1.36, v_num=2]
Epoch 0: 42%|████▏ | 396/937 [00:02<00:03, 167.88it/s, loss=1.36, v_num=2]
Epoch 0: 42%|████▏ | 397/937 [00:02<00:03, 168.04it/s, loss=1.36, v_num=2]
Epoch 0: 42%|████▏ | 397/937 [00:02<00:03, 167.99it/s, loss=1.37, v_num=2]
Epoch 0: 42%|████▏ | 397/937 [00:02<00:03, 167.88it/s, loss=1.37, v_num=2]
Epoch 0: 44%|████▍ | 414/937 [00:02<00:03, 167.86it/s, loss=1.29, v_num=2]
Epoch 0: 44%|████▍ | 414/937 [00:02<00:03, 167.94it/s, loss=1.29, v_num=2]
Epoch 0: 44%|████▍ | 415/937 [00:02<00:03, 167.88it/s, loss=1.29, v_num=2]
Epoch 0: 44%|████▍ | 415/937 [00:02<00:03, 167.97it/s, loss=1.29, v_num=2]
Epoch 0: 46%|████▌ | 433/937 [00:02<00:02, 168.16it/s, loss=1.37, v_num=2]
Epoch 0: 46%|████▌ | 433/937 [00:02<00:02, 168.23it/s, loss=1.37, v_num=2]
Epoch 0: 48%|████▊ | 451/937 [00:02<00:02, 168.39it/s, loss=1.55, v_num=2]
Epoch 0: 48%|████▊ | 451/937 [00:02<00:02, 168.49it/s, loss=1.55, v_num=2]
Epoch 0: 48%|████▊ | 452/937 [00:02<00:02, 168.51it/s, loss=1.55, v_num=2]
Epoch 0: 48%|████▊ | 452/937 [00:02<00:02, 168.39it/s, loss=1.55, v_num=2]
Epoch 0: 48%|████▊ | 452/937 [00:02<00:02, 168.49it/s, loss=1.57, v_num=2]
Epoch 0: 48%|████▊ | 452/937 [00:02<00:02, 168.37it/s, loss=1.57, v_num=2]
Epoch 0: 50%|█████ | 470/937 [00:02<00:02, 168.57it/s, loss=1.87, v_num=2]
Epoch 0: 50%|█████ | 470/937 [00:02<00:02, 168.64it/s, loss=1.87, v_num=2]
Epoch 0: 50%|█████ | 471/937 [00:02<00:02, 168.66it/s, loss=1.87, v_num=2]
Epoch 0: 50%|█████ | 471/937 [00:02<00:02, 168.58it/s, loss=1.87, v_num=2]
Epoch 0: 50%|█████ | 471/937 [00:02<00:02, 168.65it/s, loss=1.87, v_num=2]
Epoch 0: 52%|█████▏ | 489/937 [00:02<00:02, 168.85it/s, loss=1.58, v_num=2]
Epoch 0: 52%|█████▏ | 489/937 [00:02<00:02, 168.90it/s, loss=1.58, v_num=2]
Epoch 0: 52%|█████▏ | 490/937 [00:02<00:02, 168.92it/s, loss=1.58, v_num=2]
Epoch 0: 52%|█████▏ | 490/937 [00:02<00:02, 168.86it/s, loss=1.56, v_num=2]
Epoch 0: 52%|█████▏ | 490/937 [00:02<00:02, 168.90it/s, loss=1.56, v_num=2]
Epoch 0: 54%|█████▍ | 508/937 [00:03<00:02, 169.07it/s, loss=1.64, v_num=2]
Epoch 0: 54%|█████▍ | 508/937 [00:03<00:02, 169.12it/s, loss=1.64, v_num=2]
Epoch 0: 56%|█████▌ | 526/937 [00:03<00:02, 169.28it/s, loss=1.61, v_num=2]
Epoch 0: 56%|█████▌ | 526/937 [00:03<00:02, 169.34it/s, loss=1.61, v_num=2]
Epoch 0: 56%|█████▌ | 527/937 [00:03<00:02, 169.29it/s, loss=1.58, v_num=2]
Epoch 0: 56%|█████▌ | 527/937 [00:03<00:02, 169.35it/s, loss=1.58, v_num=2]
Epoch 0: 58%|█████▊ | 545/937 [00:03<00:02, 169.52it/s, loss=1.51, v_num=2]
Epoch 0: 58%|█████▊ | 545/937 [00:03<00:02, 169.56it/s, loss=1.51, v_num=2]
Epoch 0: 60%|██████ | 563/937 [00:03<00:02, 169.70it/s, loss=1.56, v_num=2]
Epoch 0: 60%|██████ | 563/937 [00:03<00:02, 169.75it/s, loss=1.56, v_num=2]
Epoch 0: 60%|██████ | 564/937 [00:03<00:02, 169.71it/s, loss=1.55, v_num=2]
Epoch 0: 60%|██████ | 564/937 [00:03<00:02, 169.77it/s, loss=1.55, v_num=2]
Epoch 0: 62%|██████▏ | 582/937 [00:03<00:02, 169.91it/s, loss=1.63, v_num=2]
Epoch 0: 62%|██████▏ | 582/937 [00:03<00:02, 169.92it/s, loss=1.63, v_num=2]
Epoch 0: 64%|██████▍ | 600/937 [00:03<00:01, 170.08it/s, loss=1.64, v_num=2]
Epoch 0: 64%|██████▍ | 600/937 [00:03<00:01, 170.09it/s, loss=1.64, v_num=2]
Epoch 0: 64%|██████▍ | 601/937 [00:03<00:01, 170.09it/s, loss=1.65, v_num=2]
Epoch 0: 64%|██████▍ | 601/937 [00:03<00:01, 170.05it/s, loss=1.65, v_num=2]
Epoch 0: 66%|██████▌ | 619/937 [00:03<00:01, 170.19it/s, loss=1.59, v_num=2]
Epoch 0: 66%|██████▌ | 619/937 [00:03<00:01, 170.22it/s, loss=1.59, v_num=2]
Epoch 0: 68%|██████▊ | 637/937 [00:03<00:01, 170.34it/s, loss=1.61, v_num=2]
Epoch 0: 68%|██████▊ | 637/937 [00:03<00:01, 170.37it/s, loss=1.61, v_num=2]
Epoch 0: 68%|██████▊ | 638/937 [00:03<00:01, 170.35it/s, loss=1.61, v_num=2]
Epoch 0: 68%|██████▊ | 638/937 [00:03<00:01, 170.38it/s, loss=1.61, v_num=2]
Epoch 0: 70%|██████▉ | 655/937 [00:03<00:01, 170.40it/s, loss=1.57, v_num=2]
Epoch 0: 70%|███████ | 656/937 [00:03<00:01, 170.44it/s, loss=1.54, v_num=2]
Epoch 0: 70%|███████ | 656/937 [00:03<00:01, 170.41it/s, loss=1.54, v_num=2]
Epoch 0: 72%|███████▏ | 674/937 [00:03<00:01, 170.56it/s, loss=1.62, v_num=2]
Epoch 0: 72%|███████▏ | 674/937 [00:03<00:01, 170.60it/s, loss=1.62, v_num=2]
Epoch 0: 72%|███████▏ | 675/937 [00:03<00:01, 170.58it/s, loss=1.62, v_num=2]
Epoch 0: 72%|███████▏ | 675/937 [00:03<00:01, 170.60it/s, loss=1.62, v_num=2]
Epoch 0: 72%|███████▏ | 675/937 [00:03<00:01, 170.57it/s, loss=1.62, v_num=2]
Epoch 0: 74%|███████▍ | 692/937 [00:04<00:01, 170.53it/s, loss=1.72, v_num=2]
Epoch 0: 74%|███████▍ | 693/937 [00:04<00:01, 170.58it/s, loss=1.7, v_num=2]
Epoch 0: 74%|███████▍ | 693/937 [00:04<00:01, 170.53it/s, loss=1.7, v_num=2]
Epoch 0: 76%|███████▌ | 710/937 [00:04<00:01, 170.50it/s, loss=1.87, v_num=2]
Epoch 0: 76%|███████▌ | 711/937 [00:04<00:01, 170.55it/s, loss=1.86, v_num=2]
Epoch 0: 76%|███████▌ | 711/937 [00:04<00:01, 170.50it/s, loss=1.86, v_num=2]
Epoch 0: 78%|███████▊ | 729/937 [00:04<00:01, 170.55it/s, loss=1.85, v_num=2]
Epoch 0: 78%|███████▊ | 729/937 [00:04<00:01, 170.59it/s, loss=1.85, v_num=2]
Epoch 0: 80%|███████▉ | 747/937 [00:04<00:01, 170.64it/s, loss=1.84, v_num=2]
Epoch 0: 80%|███████▉ | 747/937 [00:04<00:01, 170.68it/s, loss=1.84, v_num=2]
Epoch 0: 80%|███████▉ | 748/937 [00:04<00:01, 170.68it/s, loss=1.87, v_num=2]
Epoch 0: 80%|███████▉ | 748/937 [00:04<00:01, 170.64it/s, loss=1.87, v_num=2]
^C/home/ubuntu/anaconda3/envs/plt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:726: UserWarning: Detected KeyboardInterrupt, attempting graceful shutdown...
rank_zero_warn("Detected KeyboardInterrupt, attempting graceful shutdown...")
Epoch 0: 82%|████████▏ | 766/937 [00:04<00:01, 170.72it/s, loss=1.88, v_num=2]
Epoch 0: 82%|████████▏ | 766/937 [00:04<00:01, 170.75it/s, loss=1.88, v_num=2]
Epoch 0: 84%|████████▎ | 784/937 [00:04<00:00, 170.80it/s, loss=1.81, v_num=2]
Epoch 0: 84%|████████▎ | 784/937 [00:04<00:00, 170.84it/s, loss=1.81, v_num=2]
Epoch 0: 86%|████████▌ | 802/937 [00:04<00:00, 170.91it/s, loss=1.95, v_num=2]
Epoch 0: 86%|████████▌ | 802/937 [00:04<00:00, 170.94it/s, loss=1.95, v_num=2]
Epoch 0: 86%|████████▌ | 803/937 [00:04<00:00, 170.91it/s, loss=1.95, v_num=2]
Epoch 0: 86%|████████▌ | 803/937 [00:04<00:00, 170.93it/s, loss=1.95, v_num=2]
Epoch 0: 86%|████████▌ | 803/937 [00:04<00:00, 170.90it/s, loss=1.95, v_num=2]
