yolov9 icon indicating copy to clipboard operation
yolov9 copied to clipboard

ISSUE: IndexError: list index out of range - while training 'YOLOv9-e' on custom dataset.

Open pranavdurai10 opened this issue 11 months ago • 18 comments

I'm trying to train this yolov9-e.pt model on a custom dataset.

Here's my command-line argument:

!python train_dual.py \ --batch 64 --epochs 25 --img 640 --device 0 --min-items 0 --close-mosaic 15 --workers 4\ --data /content/yolov9/SkyFusion-YOLOv9/data.yaml \ --weights {HOME}/weights/yolov9-e.pt \ --cfg models/detect/yolov9-e.yaml \ --hyp hyp.scratch-high.yaml

However, I keep running into this IndexError: list index out of rangeerror.

Here's what the traceback looks like:

yolov9-e summary: 1475 layers, 69410930 parameters, 69410898 gradients, 244.9 GFLOPs

Transferred 2160/2172 items from {HOME}/weights/yolov9-e.pt AMP: checks passed ✅ optimizer: SGD(lr=0.01) with parameter groups 356 weight(decay=0.0), 375 weight(decay=0.0005), 373 bias albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8)) train: Scanning /content/yolov9/SkyFusion-YOLOv9/train/labels.cache... 2094 images, 0 backgrounds, 0 corrupt: 100% 2094/2094 [00:00<?, ?it/s] val: Scanning /content/yolov9/SkyFusion-YOLOv9/valid/labels.cache... 449 images, 0 backgrounds, 0 corrupt: 100% 449/449 [00:00<?, ?it/s] Plotting labels to runs/train/exp18/labels.jpg... Image sizes 640 train, 640 val Using 2 dataloader workers Logging results to runs/train/exp18 Starting training for 25 epochs...

  Epoch    GPU_mem   box_loss   cls_loss   dfl_loss  Instances       Size

0% 0/33 [00:00<?, ?it/s] Traceback (most recent call last): File "/content/yolov9/train_dual.py", line 644, in main(opt) File "/content/yolov9/train_dual.py", line 538, in main train(opt.hyp, opt, device, callbacks) File "/content/yolov9/train_dual.py", line 288, in train for i, (imgs, targets, paths, _) in pbar: # batch ------------------------------------------------------------- File "/usr/local/lib/python3.10/dist-packages/tqdm/std.py", line 1181, in iter for obj in iterable: File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 630, in next data = self._next_data() File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1345, in _next_data return self._process_data(data) File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1371, in _process_data data.reraise() File "/usr/local/lib/python3.10/dist-packages/torch/_utils.py", line 694, in reraise raise exception IndexError: Caught IndexError in DataLoader worker process 0. Original Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop data = fetcher.fetch(index) File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py", line 51, in data = [self.dataset[idx] for idx in possibly_batched_index] File "/content/yolov9/utils/dataloaders.py", line 656, in getitem img, labels = self.load_mosaic(index) File "/content/yolov9/utils/dataloaders.py", line 791, in load_mosaic img4, labels4, segments4 = copy_paste(img4, labels4, segments4, p=self.hyp['copy_paste']) File "/content/yolov9/utils/augmentations.py", line 248, in copy_paste l, box, s = labels[j], boxes[j], segments[j] IndexError: list index out of range

Can someone please assist with this?

pranavdurai10 avatar Mar 05 '24 06:03 pranavdurai10

I am also getting the same error I tried the default dataset that was loaded on the yolov9 google collab which is: project = rf.workspace("roboflow-jvuqo").project("football-players-detection-3zvbc") version = project.version(8) dataset = version.download("yolov9") and the code: !python train_dual.py \ --batch 64 --epochs 25 --img 640 --device 0 --min-items 0 --close-mosaic 15 --workers 4\ --data /content/yolov9/SkyFusion-YOLOv9/data.yaml \ --weights {HOME}/weights/yolov9-e.pt \ --cfg models/detect/yolov9-e.yaml \ --hyp hyp.scratch-high.yaml works perfectly but when I use this dataset: project = rf.workspace("zr-testing").project("zrv2") version = project.version(32) dataset = version.download("yolov9") I get the Same error as @pranavdurai10

parasmaisalunkhe avatar Mar 05 '24 23:03 parasmaisalunkhe

i have the same problem :(

 Epoch    GPU_mem   box_loss   cls_loss   dfl_loss  Instances       Size

0%| | 0/1283 00:00 Traceback (most recent call last): File "C:\Users\kitag\PycharmProjects\AI\Lab\Yolov9\train.py", line 636, in main(opt) File "C:\Users\kitag\PycharmProjects\AI\Lab\Yolov9\train.py", line 530, in main train(opt.hyp, opt, device, callbacks) File "C:\Users\kitag\PycharmProjects\AI\Lab\Yolov9\train.py", line 279, in train for i, (imgs, targets, paths, _) in pbar: # batch ------------------------------------------------------------- File "C:\Users\kitag\anaconda3\envs\Lib\site-packages\tqdm\std.py", line 1195, in iter for obj in iterable: File "C:\Users\kitag\PycharmProjects\AI\Lab\Yolov9\utils\dataloaders.py", line 170, in iter yield next(self.iterator) ^^^^^^^^^^^^^^^^^^^ File "C:\Users\kitag\anaconda3\envs\Lib\site-packages\torch\utils\data\dataloader.py", line 630, in next data = self._next_data() ^^^^^^^^^^^^^^^^^ File "C:\Users\kitag\anaconda3\envs\Lib\site-packages\torch\utils\data\dataloader.py", line 1345, in _next_data return self._process_data(data) ^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\kitag\anaconda3\envs\Lib\site-packages\torch\utils\data\dataloader.py", line 1371, in _process_data data.reraise() File "C:\Users\kitag\anaconda3\envs\Lib\site-packages\torch_utils.py", line 694, in reraise raise exception IndexError: Caught IndexError in DataLoader worker process 0. Original Traceback (most recent call last): File "C:\Users\kitag\anaconda3\envs\Lib\site-packages\torch\utils\data_utils\worker.py", line 308, in _worker_loop data = fetcher.fetch(index) ^^^^^^^^^^^^^^^^^^^^ File "C:\Users\kitag\anaconda3\envs\Lib\site-packages\torch\utils\data_utils\fetch.py", line 51, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\kitag\anaconda3\envs\Lib\site-packages\torch\utils\data_utils\fetch.py", line 51, in data = [self.dataset[idx] for idx in possibly_batched_index] ~~~~~~~~~~~~^^^^^ File "C:\Users\kitag\PycharmProjects\AI\Lab\Yolov9\utils\dataloaders.py", line 656, in getitem img, labels = self.load_mosaic(index) ^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\kitag\PycharmProjects\AI\Lab\Yolov9\utils\dataloaders.py", line 791, in load_mosaic img4, labels4, segments4 = copy_paste(img4, labels4, segments4, p=self.hyp['copy_paste']) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\kitag\PycharmProjects\AI\Lab\Yolov9\utils\augmentations.py", line 248, in copy_paste l, box, s = labels[j], boxes[j], segments[j] ~~~~~~~~^^^ IndexError: list index out of range

Kitagawayyds avatar Mar 06 '24 08:03 Kitagawayyds

@Kitagawayyds @parasmaisalunkhe - Please go ahead with Ultralytics implementation of YOLOv9.

Link: https://docs.ultralytics.com/models/yolov9/

pranavdurai10 avatar Mar 06 '24 08:03 pranavdurai10

@Kitagawayyds @parasmaisalunkhe - Please go ahead with Ultralytics implementation of YOLOv9.

Link: https://docs.ultralytics.com/models/yolov9/

is this problem solved? (in Ultralytics)

Kitagawayyds avatar Mar 06 '24 08:03 Kitagawayyds

@Kitagawayyds - Yes, I'm able to train both YOLOv9-C and E models on a custom dataset.

pranavdurai10 avatar Mar 06 '24 10:03 pranavdurai10

I solved my problem basically in roboflow I added Resize: Stretch to 640x640 to my Preproccesing of my dataset which works with the: !python train_dual.py \ --batch 64 --epochs 25 --img 640 --device 0 --min-items 0 --close-mosaic 15 --workers 4\ --data /content/yolov9/SkyFusion-YOLOv9/data.yaml \ --weights {HOME}/weights/yolov9-e.pt \ --cfg models/detect/yolov9-e.yaml \ --hyp hyp.scratch-high.yaml and I thinik the img flag represents 640 by 640 so it works now

parasmaisalunkhe avatar Mar 07 '24 14:03 parasmaisalunkhe

@parasmaisalunkhe - in that way, my dataset already had 640x640 images.

It still didn't work.

pranavdurai10 avatar Mar 07 '24 14:03 pranavdurai10

thanks the ultralytics documentation works now

parasmaisalunkhe avatar Mar 07 '24 15:03 parasmaisalunkhe

thanks the ultralytics documentation works now

bro.. could you tell me how did you resolved that issue on IndexError: list index out of range

PranethReddy avatar Mar 17 '24 05:03 PranethReddy

same issue here I am using this dataset which already has, Resize: Stretch to 640x640 rf = Roboflow(api_key="") project = rf.workspace("university-of-southeastern-philippines-cnl9c").project("license-plate-detection-merged-projects") version = project.version(3) dataset = version.download("yolov9")

!python train.py
--batch 16 --epochs 25 --img 640 --device 0 --min-items 0 --close-mosaic 15
--data {dataset.location}/data.yaml
--weights {HOME}/weights/yolov9-c.pt
--cfg models/detect/yolov9.yaml
--hyp hyp.scratch-high.yaml please check the error

Narenk9 avatar Mar 17 '24 12:03 Narenk9

@parasmaisalunkhe, could you kindly explain how you resized the dataset? I'm not familiar with how to do it using the YAML format from Roboflow.

IsraaSaede avatar Mar 19 '24 00:03 IsraaSaede

modify hyp.scratch-high.yaml old copy_paste: 0.3 # segment copy-paste (probability) new copy_paste: 0.1 or 0.0

googzzgsee avatar Mar 20 '24 07:03 googzzgsee

@googzzgsee not working

abdulkadrtr avatar Mar 21 '24 13:03 abdulkadrtr

Modify the copy_paste parameter in hyp.scratch-high.yaml to 0.0 helped me solve the problem. image

ytl0623 avatar Mar 22 '24 07:03 ytl0623

tysm @ytl0623 it works for me

parasmaisalunkhe avatar Mar 25 '24 21:03 parasmaisalunkhe

Modify the copy_paste parameter in hyp.scratch-high.yaml to 0.0 helped me solve the problem. image

its working but can you explain the intution behind this ? why is that so ?

Jacko760 avatar Apr 16 '24 15:04 Jacko760

Starting training for 5 epochs...

  Epoch    GPU_mem   box_loss   cls_loss   dfl_loss  Instances       Size
    0/4      1.85G      2.066      3.152      2.312          4        640:   0%|          | 0/303 00:02WARNING ⚠️ TensorBoard graph visualization failure Only tensors, lists, tuples of tensors, or dictionary of tensors can be output from traced functions
    0/4      2.89G      2.106      4.011      2.589          1        640: 100%|██████████| 303/303 04:18
             Class     Images  Instances          P          R      mAP50   mAP50-95:   0%|          | 0/11 00:00si: 0, len(paths): 2, len(shapes): 2

si: 1, len(paths): 2, len(shapes): 2 Class Images Instances P R mAP50 mAP50-95: 0%| | 0/11 00:00 Traceback (most recent call last): File "train_dual.py", line 644, in main(opt) File "train_dual.py", line 538, in main train(opt.hyp, opt, device, callbacks) File "train_dual.py", line 356, in train results, maps, _ = validate.run(data_dict, File "/data/hxq/anaconda3/envs/YOLOv9/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "/data/hxq/work/Comparison-model/yolov9/val_dual.py", line 228, in run path, shape = Path(paths[si]), shapes[si][0] IndexError: list index out of range 还是无法解决

hxq11 avatar Jun 28 '24 09:06 hxq11

Modify the copy_paste parameter in hyp.scratch-high.yaml to 0.0 helped me solve the problem. image

it is working..thank you

Varun3713-creator avatar Jun 28 '24 11:06 Varun3713-creator