Multiple train/val/test dirs

[knoppmyth]

I thought I saw where this was possible or am I mistaken? Is there support for multiple image dirs?

train: - /datasets/V1/images/train # train images (relative to 'path') - /datasets/V2/images/train

Thanks, km

[mtjhl]

Sorry, currently this project does not support multi directories, we will consider to add this funciton.

[knoppmyth]

Thanks for the response and consideration.

[shensheng272]

For now, we merge PR about recursive train/val folder, might alleviate the problem.

[knoppmyth]

Thanks for the update. Unfortunately, the PR doesn't seem to add this feature. WIth a YAML file formatted like the above, I get the following error: Traceback (most recent call last): File "/usr/src/app/tools/train.py", line 126, in <module> main(args) File "/usr/src/app/tools/train.py", line 111, in main trainer = Trainer(args, cfg, device) File "/usr/src/app/yolov6/core/engine.py", line 48, in __init__ self.train_loader, self.val_loader = self.get_data_loader(args, cfg, self.data_dict) File "/usr/src/app/yolov6/core/engine.py", line 293, in get_data_loader train_loader = create_dataloader(train_path, args.img_size, args.batch_size // args.world_size, grid_size, File "/usr/src/app/yolov6/data/data_load.py", line 41, in create_dataloader dataset = TrainValDataset( File "/usr/src/app/yolov6/data/datasets.py", line 66, in __init__ self.img_paths, self.labels = self.get_imgs_labels(self.img_dir) File "/usr/src/app/yolov6/data/datasets.py", line 214, in get_imgs_labels assert osp.exists(img_dir), f"{img_dir} is an invalid directory path!" File "/usr/lib/python3.10/genericpath.py", line 19, in exists os.stat(path) TypeError: stat: path should be string, bytes, os.PathLike or integer, not list If I format it like this: train: /usr/src/datasets/Hand/V1/images/train # train images (relative to 'path') /usr/src/datasets/Hand/V2/images/train

I get this error: Traceback (most recent call last): File "/usr/src/app/tools/train.py", line 126, in <module> main(args) File "/usr/src/app/tools/train.py", line 111, in main trainer = Trainer(args, cfg, device) File "/usr/src/app/yolov6/core/engine.py", line 46, in __init__ self.data_dict = load_yaml(args.data_path) File "/usr/src/app/yolov6/utils/events.py", line 23, in load_yaml data_dict = yaml.safe_load(f) File "/usr/lib/python3.10/site-packages/yaml/__init__.py", line 125, in safe_load return load(stream, SafeLoader) File "/usr/lib/python3.10/site-packages/yaml/__init__.py", line 81, in load return loader.get_single_data() File "/usr/lib/python3.10/site-packages/yaml/constructor.py", line 49, in get_single_data node = self.get_single_node() File "/usr/lib/python3.10/site-packages/yaml/composer.py", line 36, in get_single_node document = self.compose_document() File "/usr/lib/python3.10/site-packages/yaml/composer.py", line 55, in compose_document node = self.compose_node(None, None) File "/usr/lib/python3.10/site-packages/yaml/composer.py", line 84, in compose_node node = self.compose_mapping_node(anchor) File "/usr/lib/python3.10/site-packages/yaml/composer.py", line 127, in compose_mapping_node while not self.check_event(MappingEndEvent): File "/usr/lib/python3.10/site-packages/yaml/parser.py", line 98, in check_event self.current_event = self.state() File "/usr/lib/python3.10/site-packages/yaml/parser.py", line 438, in parse_block_mapping_key raise ParserError("while parsing a block mapping", self.marks[-1], yaml.parser.ParserError: while parsing a block mapping in "../data/hand.yaml", line 1, column 1 expected <block end>, but found '<scalar>' in "../data/hand.yaml", line 3, column 8 Am I formatting it wrong? If so, what is the correct format?

[shensheng272]

For now, we merge PR about recursive train/val folder, might alleviate the problem.

@knoppmyth The PR support you set /usr/src/datasets/Hand to search for /usr/src/datasets/Hand/V1/images/train and /usr/src/datasets/Hand/V2/images/train . (But it will search /usr/src/datasets/Hand/V2/images/val or other image under your path) If on ubuntu , you could use ln -s to create link to avoid the copy-paste.

[knoppmyth]

@shensheng272 Thanks for the update. Apologies for the late reply. Unfortunately, /usr/src/datasets/Hand doesn't work as I'd expect.

Here is is my YAML file: `path: /usr/src/datasets/Hand # dataset root dir train: /usr/src/datasets/Hand # train images (relative to 'path') val: /usr/src/datasets/Hand # val images (relative to 'path') test: # test images (optional)

nc: 1 # number of classes names: ['Hand'] # class names`

Here is the resulting error: Image dir is: /usr/src/datasets/Hand Train: Checking formats of images with 8 process(es): 0 image(s) corrupted: 100%|████████████████████| 251/251 [00:00<00:00, 8199.46it/s] Traceback (most recent call last): File "/usr/src/app/tools/train.py", line 126, in <module> main(args) File "/usr/src/app/tools/train.py", line 111, in main trainer = Trainer(args, cfg, device) File "/usr/src/app/yolov6/core/engine.py", line 48, in __init__ self.train_loader, self.val_loader = self.get_data_loader(args, cfg, self.data_dict) File "/usr/src/app/yolov6/core/engine.py", line 293, in get_data_loader train_loader = create_dataloader(train_path, args.img_size, args.batch_size // args.world_size, grid_size, File "/usr/src/app/yolov6/data/data_load.py", line 41, in create_dataloader dataset = TrainValDataset( File "/usr/src/app/yolov6/data/datasets.py", line 66, in __init__ self.img_paths, self.labels = self.get_imgs_labels(self.img_dir) File "/usr/src/app/yolov6/data/datasets.py", line 275, in get_imgs_labels assert osp.exists(label_dir), f"{label_dir} is an invalid directory path!" AssertionError: /usr/src/labels/Hand is an invalid directory path! It seems to expect the labels in it's own separate directory instead of under the "Hands". If I symlink ln -s /usr/src/datasets /usr/src/labels it does work.

While workable, it is not intuitive as when I have /usr/src/datasets/Hand/V1/images/train set it is able to properly traverse dirs to find the labels.

[knoppmyth]

After reviewing the code, a simple solution would be to just make label_dir = image_dir on line 268 in dataset.py. However if we actually want to check for a "labels" dir, it won't work multiple dataset dirs. I have a solution in mind where the code actually checks for an "labels" dir that should would with one dir or multiple. I'll write up the change and submit a PR as soon as I can.

[knoppmyth]

@shensheng272 Sumitted PR #502