FEDOT icon indicating copy to clipboard operation
FEDOT copied to clipboard

Image processing refactor

Open Dreamlone opened this issue 2 years ago • 0 comments

The alternative title for the future PR on this issue is "Batch-support implementation for image processing"

FEDOT currently has a separate DataTypesEnum.image data type. It loads all matrices into an array and then feeds some matrices in batches for CNN training. However, for most datasets with images, it is impossible to keep all matrices in RAM at the same time. Therefore, my suggestion is to create a new DataTypesEnum.image_paths data type, or modify an existing one.

The table will have something like the following structure

features target
D:/dataset_name/train/first_features.png 1
D:/dataset_name/train/second_features.png 0

or the following structure for segmentation task for example:

features target
D:/dataset_name/train/first_features.png D:/dataset_name/train/first_target.png
D:/dataset_name/train/second_features.png D:/dataset_name/train/second_target.png

Most likely, it is necessary to implement auxiliary classes that will load data into the neural network by parts (DataLoaders)

Dreamlone avatar Apr 07 '22 15:04 Dreamlone