PaddleClas icon indicating copy to clipboard operation
PaddleClas copied to clipboard

Published 20 hours ago • verified publisher icon PaddlePaddle

ColorJitter可以直接配置使用吗

Open suhangbo opened this issue 1 year ago • 2 comments

对图像数据进行随机调整图像的亮度、对比度、饱和度和色调配置如下： transform_ops： - ColorJitter: brightness: 0.4 contrast: 0.4 saturation: 0.4 hue: 0.4 paddle版本：release/2.3 以上配置可直接使用吗

suhangbo avatar Jul 11 '22 08:07 suhangbo

对应代码在 https://github.com/PaddlePaddle/PaddleClas/blob/release%2F2.3/ppcls/data/preprocess/ops/operators.py#L370 可以使用，但是建议加一个参数prob控制，即以prob的概率进行colorjitter变换，否则保持不变。

HydrogenSulfate avatar Jul 11 '22 10:07 HydrogenSulfate

对应代码在 https://github.com/PaddlePaddle/PaddleClas/blob/release%2F2.3/ppcls/data/preprocess/ops/operators.py#L370 可以使用，但是建议加一个参数prob控制，即以prob的概率进行colorjitter变换，否则保持不变。

是否适用于多标签训练呢

colorjitter只是一种数据增强手段，一般不受具体任务的限制，而是要看整体数据的情况，如果你数据中同一种类别的图像，光线、色彩、对比度变化幅度比较大，那么你可以选择加入colorjitter提高模型的泛化能力。

HydrogenSulfate avatar Jul 11 '22 11:07 HydrogenSulfate

对应代码在 https://github.com/PaddlePaddle/PaddleClas/blob/release%2F2.3/ppcls/data/preprocess/ops/operators.py#L370 可以使用，但是建议加一个参数prob控制，即以prob的概率进行colorjitter变换，否则保持不变。

我加上了colorjitter，prob参数为2,训练失败了，不加这个参数可以正常训练，报错如下： [2023/02/27 11:27:54] ppcls ERROR: Exception occured when parse line: ./dataset/Inshop/Inshop/Img/img/ALL/train/2678/47_2_side.jpg with msg: Cannot handle this data type: (1, 1), |S16212 [2023/02/27 11:27:54] ppcls ERROR: Exception occured when parse line: ./dataset/Inshop/Inshop/Img/img/ALL/train/5104/01_4_full.jpg with msg: Cannot handle this data type: (1, 1), |S14276 [2023/02/27 11:27:54] ppcls ERROR: Exception occured when parse line: ./dataset/Inshop/Inshop/Img/img/ALL/train/4125/01_3_back.jpg with msg: Cannot handle this data type: (1, 1), |S14103 [2023/02/27 11:27:54] ppcls ERROR: Exception occured when parse line: ./dataset/Inshop/Inshop/Img/img/ALL/train/1158/02_3_back.jpg with msg: Cannot handle this data type: (1, 1), |S32541 [2023/02/27 11:27:54] ppcls ERROR: Exception occured when parse line: ./dataset/Inshop/Inshop/Img/img/ALL/train/1839/02_2_side.jpg with msg: Cannot handle this data type: (1, 1), |S12800 [2023/02/27 11:27:54] ppcls ERROR: Exception occured when parse line: ./dataset/Inshop/Inshop/Img/img/ALL/train/4982/02_2_side.jpg with msg: Cannot handle this data type: (1, 1), |S15727 [2023/02/27 11:27:54] ppcls ERROR: Exception occured when parse line: ./dataset/Inshop/Inshop/Img/img/ALL/train/3284/03_1_front.jpg with msg: Cannot handle this data type: (1, 1), |S13578 [2023/02/27 11:27:54] ppcls ERROR: Exception occured when parse line: ./dataset/Inshop/Inshop/Img/img/ALL/train/5310/07_3_back.jpg with msg: Cannot handle this data type: (1, 1), |S12865 Process Process-4: Process Process-1: Traceback (most recent call last): File "tools/train.py", line 32, in engine.train() File "/mnt/sdc/PaddleClas2/ppcls/engine/engine.py", line 356, in train Exception in thread Thread-1: Traceback (most recent call last): File "/mnt/sdc/driver/Anaconda/envs/paddle/lib/python3.7/site-packages/paddle/fluid/dataloader/dataloader_iter.py", line 623, in _get_data data = self._data_queue.get(timeout=self._timeout) File "/mnt/sdc/driver/Anaconda/envs/paddle/lib/python3.7/multiprocessing/queues.py", line 105, in get raise Empty _queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/mnt/sdc/driver/Anaconda/envs/paddle/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/mnt/sdc/driver/Anaconda/envs/paddle/lib/python3.7/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/mnt/sdc/driver/Anaconda/envs/paddle/lib/python3.7/site-packages/paddle/fluid/dataloader/dataloader_iter.py", line 536, in _thread_loop batch = self._get_data() File "/mnt/sdc/driver/Anaconda/envs/paddle/lib/python3.7/site-packages/paddle/fluid/dataloader/dataloader_iter.py", line 639, in _get_data "pids: {}".format(len(failed_workers), pids)) RuntimeError: DataLoader 4 workers exit unexpectedly, pids: 9322, 9331, 9340, 9348

self.train_epoch_func(self, epoch_id, print_batch_step)

File "/mnt/sdc/PaddleClas2/ppcls/engine/train/train.py", line 24, in train_epoch for iter_id, batch in enumerate(engine.train_dataloader): File "/mnt/sdc/driver/Anaconda/envs/paddle/lib/python3.7/site-packages/paddle/fluid/dataloader/dataloader_iter.py", line 745, in next self.reader.read_next_list()[0]) SystemError: (Fatal) Blocking queue is killed because the data reader raises an exception. [Hint: Expected killed != true, but received killed_:1 == true:1.] (at /paddle/paddle/fluid/operators/reader/blocking_queue.h:175)

I0227 11:27:59.591239 9305 tcp_store.cc:257] receive shutdown event and so quit from MasterDaemon run loop LAUNCH INFO 2023-02-27 11:28:00,050 Pod failed LAUNCH ERROR 2023-02-27 11:28:00,051 Container failed !!! Container rank 0 status failed cmd ['/mnt/sdc/driver/Anaconda/envs/paddle/bin/python3.7', '-u', 'tools/train.py', '-c', './ppcls/configs/GeneralRecognitionV2//GeneralRecognitionV2_PPLCNetV2_Inshop.yaml'] code 1 log log/workerlog.0 env {'CONDA_SHLVL': '2', 'LD_LIBRARY_PATH': '/mnt/sdc/driver/Anaconda/envs/paddle/lib/python3.7/site-packages/cv2/../../lib64:/usr/local/cuda-11.2/lib64', 'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arc=01;31:.arj=01;31:.taz=01;31:.lha=01;31:.lz4=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.tzo=01;31:.t7z=01;31:.zip=01;31:.z=01;31:.Z=01;31:.dz=01;31:.gz=01;31:.lrz=01;31:.lz=01;31:.lzo=01;31:.xz=01;31:.zst=01;31:.tzst=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.alz=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.cab=01;31:.wim=01;31:.swm=01;31:.dwm=01;31:.esd=01;31:.jpg=01;35:.jpeg=01;35:.mjpg=01;35:.mjpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.ogv=01;35:.ogx=01;35:.aac=00;36:.au=00;36:.flac=00;36:.m4a=00;36:.mid=00;36:.midi=00;36:.mka=00;36:.mp3=00;36:.mpc=00;36:.ogg=00;36:.ra=00;36:.wav=00;36:.oga=00;36:.opus=00;36:.spx=00;36:.xspf=00;36:', 'CONDA_EXE': '/mnt/sdc/driver/Anaconda/bin/conda', 'LC_MEASUREMENT': 'C.UTF-8', 'SSH_CONNECTION': '223.112.162.34 51564 192.168.0.118 22', 'LESSCLOSE': '/usr/bin/lesspipe %s %s', 'LC_PAPER': 'C.UTF-8', 'LC_MONETARY': 'C.UTF-8', 'LANG': 'C.UTF-8', 'HISTTIMEFORMAT': '%F %T root ', 'OLDPWD': '/mnt/sdc/PaddleClas2/dataset/Inshop/Inshop', 'CONDA_PREFIX': '/mnt/sdc/driver/Anaconda/envs/paddle', '_CE_M': '', 'LC_NAME': 'C.UTF-8', 'XDG_SESSION_ID': '216', 'USER': 'root', 'CONDA_PREFIX_1': '/mnt/sdc/driver/Anaconda', 'PWD': '/mnt/sdc/PaddleClas2', 'HOME': '/root', 'CONDA_PYTHON_EXE': '/mnt/sdc/driver/Anaconda/bin/python', 'SSH_CLIENT': '223.112.162.34 51564 22', 'CUDA_HOME': '/usr/local/cuda-11.2', 'XDG_DATA_DIRS': '/usr/local/share:/usr/share:/var/lib/snapd/desktop', 'CE_CONDA': '', 'LC_ADDRESS': 'C.UTF-8', 'LC_NUMERIC': 'C.UTF-8', 'CONDA_PROMPT_MODIFIER': '(paddle) ', 'SSH_TTY': '/dev/pts/0', 'MAIL': '/var/mail/root', 'TERM': 'xterm-256color', 'SHELL': '/bin/bash', 'LD_KIBRARY_PATH': '/usr/local/cuda-11.2/lib64:/usr/lib/x86_64-linux-gnu', 'CUDA_VISIBLE_DEVICES': '0,1,2,3', 'SHLVL': '1', 'LANGUAGE': 'en_US:', 'LC_TELEPHONE': 'C.UTF-8', 'LOGNAME': 'root', 'XDG_RUNTIME_DIR': '/run/user/0', 'PATH': '/mnt/sdc/driver/Anaconda/envs/paddle/bin:/mnt/sdc/driver/Anaconda/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/cuda-11.2/bin', 'LC_IDENTIFICATION': 'C.UTF-8', 'CONDA_DEFAULT_ENV': 'paddle', 'HISTSIZE': '1000', 'LESSOPEN': '| /usr/bin/lesspipe %s', 'LC_TIME': 'C.UTF-8', '': '/mnt/sdc/driver/Anaconda/envs/paddle/bin/python3.7', 'CUSTOM_DEVICE_ROOT': '', 'OMP_NUM_THREADS': '1', 'QT_QPA_PLATFORM_PLUGIN_PATH': '/mnt/sdc/driver/Anaconda/envs/paddle/lib/python3.7/site-packages/cv2/qt/plugins', 'QT_QPA_FONTDIR': '/mnt/sdc/driver/Anaconda/envs/paddle/lib/python3.7/site-packages/cv2/qt/fonts', 'POD_NAME': 'xhquot', 'PADDLE_MASTER': '127.0.1.1:59870', 'PADDLE_GLOBAL_SIZE': '4', 'PADDLE_LOCAL_SIZE': '4', 'PADDLE_GLOBAL_RANK': '0', 'PADDLE_LOCAL_RANK': '0', 'PADDLE_NNODES': '1', 'PADDLE_TRAINER_ENDPOINTS': '127.0.1.1:59871,127.0.1.1:59872,127.0.1.1:59873,127.0.1.1:59874', 'PADDLE_CURRENT_ENDPOINT': '127.0.1.1:59871', 'PADDLE_TRAINER_ID': '0', 'PADDLE_TRAINERS_NUM': '4', 'PADDLE_RANK_IN_NODE': '0', 'FLAGS_selected_gpus': '0'} LAUNCH INFO 2023-02-27 11:28:00,051 ------------------------- ERROR LOG DETAIL ------------------------- e: ./dataset/Inshop/Inshop/Img/img/ALL/train/1158/02_3_back.jpg with msg: Cannot handle this data type: (1, 1), |S32541 [2023/02/27 11:27:54] ppcls ERROR: Exception occured when parse line: ./dataset/Inshop/Inshop/Img/img/ALL/train/1839/02_2_side.jpg with msg: Cannot handle this data type: (1, 1), |S12800 [2023/02/27 11:27:54] ppcls ERROR: Exception occured when parse line: ./dataset/Inshop/Inshop/Img/img/ALL/train/4982/02_2_side.jpg with msg: Cannot handle this data type: (1, 1), |S15727 [2023/02/27 11:27:54] ppcls ERROR: Exception occured when parse line: ./dataset/Inshop/Inshop/Img/img/ALL/train/3284/03_1_front.jpg with msg: Cannot handle this data type: (1, 1), |S13578 [2023/02/27 11:27:54] ppcls ERROR: Exception occured when parse line: ./dataset/Inshop/Inshop/Img/img/ALL/train/5310/07_3_back.jpg with msg: Cannot handle this data type: (1, 1), |S12865 Process Process-4: Process Process-1: Traceback (most recent call last): File "tools/train.py", line 32, in engine.train() File "/mnt/sdc/PaddleClas2/ppcls/engine/engine.py", line 356, in train Exception in thread Thread-1: Traceback (most recent call last): File "/mnt/sdc/driver/Anaconda/envs/paddle/lib/python3.7/site-packages/paddle/fluid/dataloader/dataloader_iter.py", line 623, in _get_data data = self._data_queue.get(timeout=self._timeout) File "/mnt/sdc/driver/Anaconda/envs/paddle/lib/python3.7/multiprocessing/queues.py", line 105, in get raise Empty _queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/mnt/sdc/driver/Anaconda/envs/paddle/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/mnt/sdc/driver/Anaconda/envs/paddle/lib/python3.7/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/mnt/sdc/driver/Anaconda/envs/paddle/lib/python3.7/site-packages/paddle/fluid/dataloader/dataloader_iter.py", line 536, in _thread_loop batch = self._get_data() File "/mnt/sdc/driver/Anaconda/envs/paddle/lib/python3.7/site-packages/paddle/fluid/dataloader/dataloader_iter.py", line 639, in _get_data "pids: {}".format(len(failed_workers), pids)) RuntimeError: DataLoader 4 workers exit unexpectedly, pids: 9322, 9331, 9340, 9348

self.train_epoch_func(self, epoch_id, print_batch_step)

File "/mnt/sdc/PaddleClas2/ppcls/engine/train/train.py", line 24, in train_epoch for iter_id, batch in enumerate(engine.train_dataloader): File "/mnt/sdc/driver/Anaconda/envs/paddle/lib/python3.7/site-packages/paddle/fluid/dataloader/dataloader_iter.py", line 745, in next self.reader.read_next_list()[0]) SystemError: (Fatal) Blocking queue is killed because the data reader raises an exception. [Hint: Expected killed != true, but received killed_:1 == true:1.] (at /paddle/paddle/fluid/operators/reader/blocking_queue.h:175)

I0227 11:27:59.591239 9305 tcp_store.cc:257] receive shutdown event and so quit from MasterDaemon run loop LAUNCH INFO 2023-02-27 11:28:00,051 Exit code 1

训练配置为： image

PaulX1029 avatar Feb 27 '23 03:02 PaulX1029