PaddleOCR consistently getting Loss = nanxxx and accuracy 0.000000 even for running on 1500 epoches on arabic dataset.

trafficstars

I am trying to train PaddleOCR for arabic dataset for recognition, I am getting

I am training using this command

python -m paddle.distributed.launch --gpus '0' tools/train.py -c configs/rec/PP-OCRv3/multi_language/arabic_PP-OCRv3_rec.yml

No. of Training Samples: 95998 No. of val Samples: 10428

Here is the sample epoch output

[2023/10/02 10:05:52] ppocr INFO: epoch: [1352/1500], global_step: 2440, lr: 0.000162, acc: 0.000000, norm_edit_dis: 0.000000,CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30052 s, avg_samples: 128.0,ips: 425.93432 samples/s, eta: 1 day, 8:19:23

this is my arabic_PP-OCRv3_rec.yml

Global:
  debug: false
  use_gpu: true
  epoch_num: 500
  log_smooth_window: 20
  print_batch_step: 10
  save_model_dir: ./output/v3_arabic_mobile
  save_epoch_step: 3
  eval_batch_step: [0, 2000]
  cal_metric_during_train: true
  pretrained_model: /additional_drive/ibrar/PaddleOCR/pretrain_models/arabic/arabic_PP-OCRv3_rec_train/best_accuracy.pdparams
  checkpoints:
  save_inference_dir:
  use_visualdl: false
  infer_img: ./doc/imgs_words/arabic/ar_2.jpg
  character_dict_path: ppocr/utils/dict/arabic_dict.txt
  max_text_length: &max_text_length 25
  infer_mode: false
  use_space_char: true
  distributed: true
  save_res_path: ./output/rec/predicts_ppocrv3_arabic.txt


Optimizer:
  name: Adam
  beta1: 0.9
  beta2: 0.999
  lr:
    name: Cosine
    learning_rate: 0.00025
    warmup_epoch: 5
  regularizer:
    name: L2
    factor: 3.0e-05


Architecture:
  model_type: rec
  algorithm: SVTR
  Transform:
  Backbone:
    name: MobileNetV1Enhance
    scale: 0.5
    last_conv_stride: [1, 2]
    last_pool_type: avg
  Head:
    name: MultiHead
    head_list:
      - CTCHead:
          Neck:
            name: svtr
            dims: 64
            depth: 2
            hidden_dims: 120
            use_guide: True
          Head:
            fc_decay: 0.00001
      - SARHead:
          enc_dim: 512
          max_text_length: *max_text_length

Loss:
  name: MultiLoss
  loss_config_list:
    - CTCLoss:
    - SARLoss:

PostProcess:  
  name: CTCLabelDecode

Metric:
  name: RecMetric
  main_indicator: acc
  ignore_space: False

Train:
  dataset:
    name: SimpleDataSet
    data_dir: /additional_drive/zain/dataset/raw_data/arabic_docs_combined_caparsoft_Sep22/train/ #images_arr_updated/
    ext_op_transform_idx: 1
    label_file_list:
    - /additional_drive/zain/dataset/raw_data/arabic_docs_combined_caparsoft_Sep22/train/paddle_rec_arr_updated.txt
    transforms:
    - DecodeImage:
        img_mode: BGR
        channel_first: false
    - RecConAug:
        prob: 0.5
        ext_data_num: 2
        image_shape: [48, 320, 3]
    - RecAug:
    - MultiLabelEncode:
    - RecResizeImg:
        image_shape: [3, 48, 320]
    - KeepKeys:
        keep_keys:
        - image
        - label_ctc
        - label_sar
        - length
        - valid_ratio
  loader:
    shuffle: true
    batch_size_per_card: 128
    drop_last: true
    num_workers: 4
Eval:
  dataset:
    name: SimpleDataSet
    data_dir: /additional_drive/zain/dataset/raw_data/arabic_docs_combined_caparsoft_Sep22/val/ 
    label_file_list:
    - /additional_drive/zain/dataset/raw_data/arabic_docs_combined_caparsoft_Sep22/val/paddle_rec_arr_updated.txt
    transforms:
    - DecodeImage:
        img_mode: BGR
        channel_first: false
    - MultiLabelEncode:
    - RecResizeImg:
        image_shape: [3, 48, 320]
    - KeepKeys:
        keep_keys:
        - image
        - label_ctc
        - label_sar
        - length
        - valid_ratio
  loader:
    shuffle: false
    drop_last: false
    batch_size_per_card: 128
    num_workers: 4`

I'm attempting to train the model solely on my Arabic dataset without fine-tuning, and I'm encountering the same issue whether I use a pretrained model and fine-tune it or train it directly on my Arabic dataset.

I have attempted to resolve this issue by extensively searching through PaddleOCR's GitHub issues, and I discovered that the only suggested solution is to increase the number of epochs. Consequently, I increased the number of epochs from 200 to 1500, but unfortunately, I have not been able to resolve the issue.

Is there anyone here who can provide assistance? What I am missing?

Oct 02 '23 10:10 IbrarBabar009

@WenmuZhou @dyning @LDOUBLEV @tink2123 @MissPenguin @Topdu @Evezerest @littletomatodonkey @andyjpaddle @weisy11 @D-DanielYang @Topdu @sdcb @ZeyuChen @bingooo please help?????

I've been stuck on PaddleOCR for the past two weeks. I'd be grateful if someone could help me resolve it.

Oct 02 '23 10:10 IbrarBabar009

which version of paddleocr (not talking about venv version), the code you downloaded as zip for training what is the version of that, and also share example of your dataset, the next thing important is are you updating your dict file, this file contains all unique characters in you ground truth file

character_dict_path: ppocr/utils/dict/arabic_dict.txt

Oct 02 '23 14:10 hritikakolkar

@hritikakolkar

PaddleOCR version that I am using is 2.7.0.3 paddlepaddle-gpu : 2.4.2.post117 Python version 3.8
Dataset Directory: /additional_drive/zain/dataset/raw_data/arabic_docs_combined_caparsoft_Sep22/

Train Data:

Label File: /additional_drive/zain/dataset/raw_data/arabic_docs_combined_caparsoft_Sep22/train/paddle_rec_arr_updated.txt

images_arr_updated/0000020_0.jpeg	تاريخ
images_arr_updated/0000020_1.jpeg	سفر
images_arr_updated/0000020_2.jpeg	مكان
images_arr_updated/0000020_3.jpeg	رقم
images_arr_updated/0000020_4.jpeg	١٤٥٣/٠٥/٠١
(…….)

Images Directory: /additional_drive/zain/dataset/raw_data/arabic_docs_combined_caparsoft_Sep22/train/images_arr_updated/

  - 0000020_0.jpeg
  - 0000020_1.jpeg
  - 0000020_2.jpeg
  - 0000020_3.jpeg
  - 0000020_4.jpeg
  - ... (other images)

Same format goes for Validation directory

And this is what I have in my ground truth dict character_dict_path: ppocr/utils/dict/arabic_dict.txt

!
#
$
%
^
*
(
)
-
_
+
=
{
}
[
]
\
:
<
>
/
~
`
'
"
،
؛
؟
ء
آ
أ
ؤ
إ
ئ
ا
ب
ة
ت
ث
ج
ح
خ
د
ذ
ر
ز
س
ش
ص
ض
ط
ظ
ع
غ
ف
ق
ك
ل
م
ن
ه
و
ى
ي
٠
١
٢
٣
٤
٥
٦
٧
٨
٩
٪
٫
ٹ
پ
چ
ڈ
ژ
ک
گ
ں
ھ
ے
۴
۵
۶

Please let me if you need more information..?

Oct 03 '23 06:10 IbrarBabar009

Please share example of data. I mean like the below.

Oct 04 '23 04:10 hritikakolkar

train using this command

python -m  tools/train.py -c configs/rec/PP-OCRv3/multi_language/arabic_PP-OCRv3_rec.yml

and make distributed false In config file

Oct 05 '23 06:10 hritikakolkar

Dear @hritikakolkar It didn't solve my problem. here is the detailed explaination of how I set enviroment of PaddleOCR.

git clone https://github.com/PaddlePaddle/PaddleOCR.git

cd PaddleOCR/

conda create --name PaddleOCR python=3.7

conda activate PaddleOCR

pip install -r requirements.txt

pip install paddlepaddle-gpu

python setup.py install

After that i test an image using pretrained model downloaded from PaddleOCR github repo.

python3 tools/infer_rec.py -c configs/rec/PP-OCRv3/en_PP-OCRv3_rec.yml -o Global.pretrained_model=/additional_drive/ibrar/PaddleOCR/pretrain_models/en_PP-OCRv3_rec_train/best_accuracy.pdparams Global.infer_img=/additional_drive2/ibrar/preprocessing/test_img/colon.png Global.use_gpu=False

It accuractly recognize the image text.

ls /usr/lib | grep lib

sudo ln -s /usr/local/cuda-12.1/targets/x86_64-linux/include/libcudnn.so.8.9.1 libcudnn.so sudo ln -s /usr/local/cuda-12.1/targets/x86_64-linux/lib/libcublas.so.12.1.3.1 libcublas.so

export LD_LIBRARY_PATH=/usr/lib:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=/home/gpu1/.conda/pkgs/cudnn-7.6.5-cuda10.1_0/lib:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=/additional_drive/ibrar/PaddleOCR:$LD_LIBRARY_PATH

I have prepared my dataset of arabic language having and set the path in yml file for training PaddleOCR in my custom dataset of arabic language.

Here is the yml file

Global:
  debug: false
  use_gpu: true
  epoch_num: 500
  log_smooth_window: 20
  print_batch_step: 10
  save_model_dir: ./output/v3_arabic_mobile
  save_epoch_step: 3
  eval_batch_step: [0, 2000]
  cal_metric_during_train: true
  pretrained_model:
  checkpoints:
  save_inference_dir:
  use_visualdl: false
  infer_img: ./doc/imgs_words/arabic/ar_2.jpg
  character_dict_path: ppocr/utils/dict/arabic_dict.txt
  max_text_length: &max_text_length 25
  infer_mode: false
  use_space_char: true
  distributed: false
  save_res_path: ./output/rec/predicts_ppocrv3_arabic.txt


Optimizer:
  name: Adam
  beta1: 0.9
  beta2: 0.999
  lr:
    name: Cosine
    learning_rate: 0.001
    warmup_epoch: 5
  regularizer:
    name: L2
    factor: 3.0e-05


Architecture:
  model_type: rec
  algorithm: SVTR_LCNet
  Transform:
  Backbone:
    name: MobileNetV1Enhance
    scale: 0.5
    last_conv_stride: [1, 2]
    last_pool_type: avg
    last_pool_kernel_size: [2, 2]
  Head:
    name: MultiHead
    head_list:
      - CTCHead:
          Neck:
            name: svtr
            dims: 64
            depth: 2
            hidden_dims: 120
            use_guide: True
          Head:
            fc_decay: 0.00001
      - SARHead:
          enc_dim: 512
          max_text_length: *max_text_length

Loss:
  name: MultiLoss
  loss_config_list:
    - CTCLoss:
    - SARLoss:

PostProcess:  
  name: CTCLabelDecode

Metric:
  name: RecMetric
  main_indicator: acc
  ignore_space: False

Train:
  dataset:
    name: SimpleDataSet
    data_dir: /additional_drive2/zain/dataset/raw_data/arabic_docs_combined_caparsoft_Sep22/train/
    ext_op_transform_idx: 1
    label_file_list:
    - /additional_drive2/zain/dataset/raw_data/arabic_docs_combined_caparsoft_Sep22/train/paddle_rec_arr_updated.txt
    transforms:
    - DecodeImage:
        img_mode: BGR
        channel_first: false
    - RecConAug:
        prob: 0.5
        ext_data_num: 2
        image_shape: [48, 320, 3]
    - RecAug:
    - MultiLabelEncode:
    - RecResizeImg:
        image_shape: [3, 48, 320]
    - KeepKeys:
        keep_keys:
        - image
        - label_ctc
        - label_sar
        - length
        - valid_ratio
  loader:
    shuffle: true
    batch_size_per_card: 128
    drop_last: true
    num_workers: 4
Eval:
  dataset:
    name: SimpleDataSet
    data_dir: /additional_drive2/zain/dataset/raw_data/arabic_docs_combined_caparsoft_Sep22/val/ 
    label_file_list:
    - /additional_drive2/zain/dataset/raw_data/arabic_docs_combined_caparsoft_Sep22/val/paddle_rec_arr_updated.txt
    transforms:
    - DecodeImage:
        img_mode: BGR
        channel_first: false
    - MultiLabelEncode:
    - RecResizeImg:
        image_shape: [3, 48, 320]
    - KeepKeys:
        keep_keys:
        - image
        - label_ctc
        - label_sar
        - length
        - valid_ratio
  loader:
    shuffle: false
    drop_last: false
    batch_size_per_card: 128
    num_workers: 4

python3 tools/train.py -c configs/rec/PP-OCRv3/multi_language/arabic_PP-OCRv3_rec.yml

and here is the ouput I have, still loss : nanxxx

[2023/10/06 13:21:14] ppocr INFO: Architecture : 
[2023/10/06 13:21:14] ppocr INFO:     Backbone : 
[2023/10/06 13:21:14] ppocr INFO:         last_conv_stride : [1, 2]
[2023/10/06 13:21:14] ppocr INFO:         last_pool_kernel_size : [2, 2]
[2023/10/06 13:21:14] ppocr INFO:         last_pool_type : avg
[2023/10/06 13:21:14] ppocr INFO:         name : MobileNetV1Enhance
[2023/10/06 13:21:14] ppocr INFO:         scale : 0.5
[2023/10/06 13:21:14] ppocr INFO:     Head : 
[2023/10/06 13:21:14] ppocr INFO:         head_list : 
[2023/10/06 13:21:14] ppocr INFO:             CTCHead : 
[2023/10/06 13:21:14] ppocr INFO:                 Head : 
[2023/10/06 13:21:14] ppocr INFO:                     fc_decay : 1e-05
[2023/10/06 13:21:14] ppocr INFO:                 Neck : 
[2023/10/06 13:21:14] ppocr INFO:                     depth : 2
[2023/10/06 13:21:14] ppocr INFO:                     dims : 64
[2023/10/06 13:21:14] ppocr INFO:                     hidden_dims : 120
[2023/10/06 13:21:14] ppocr INFO:                     name : svtr
[2023/10/06 13:21:14] ppocr INFO:                     use_guide : True
[2023/10/06 13:21:14] ppocr INFO:             SARHead : 
[2023/10/06 13:21:14] ppocr INFO:                 enc_dim : 512
[2023/10/06 13:21:14] ppocr INFO:                 max_text_length : 25
[2023/10/06 13:21:14] ppocr INFO:         name : MultiHead
[2023/10/06 13:21:14] ppocr INFO:     Transform : None
[2023/10/06 13:21:14] ppocr INFO:     algorithm : SVTR_LCNet
[2023/10/06 13:21:14] ppocr INFO:     model_type : rec
[2023/10/06 13:21:14] ppocr INFO: Eval : 
[2023/10/06 13:21:14] ppocr INFO:     dataset : 
[2023/10/06 13:21:14] ppocr INFO:         data_dir : /additional_drive2/zain/dataset/raw_data/arabic_docs_combined_caparsoft_Sep22/val/
[2023/10/06 13:21:14] ppocr INFO:         label_file_list : ['/additional_drive2/zain/dataset/raw_data/arabic_docs_combined_caparsoft_Sep22/val/paddle_rec_arr_updated.txt']
[2023/10/06 13:21:14] ppocr INFO:         name : SimpleDataSet
[2023/10/06 13:21:14] ppocr INFO:         transforms : 
[2023/10/06 13:21:14] ppocr INFO:             DecodeImage : 
[2023/10/06 13:21:14] ppocr INFO:                 channel_first : False
[2023/10/06 13:21:14] ppocr INFO:                 img_mode : BGR
[2023/10/06 13:21:14] ppocr INFO:             MultiLabelEncode : None
[2023/10/06 13:21:14] ppocr INFO:             RecResizeImg : 
[2023/10/06 13:21:14] ppocr INFO:                 image_shape : [3, 48, 320]
[2023/10/06 13:21:14] ppocr INFO:             KeepKeys : 
[2023/10/06 13:21:14] ppocr INFO:                 keep_keys : ['image', 'label_ctc', 'label_sar', 'length', 'valid_ratio']
[2023/10/06 13:21:14] ppocr INFO:     loader : 
[2023/10/06 13:21:14] ppocr INFO:         batch_size_per_card : 128
[2023/10/06 13:21:14] ppocr INFO:         drop_last : False
[2023/10/06 13:21:14] ppocr INFO:         num_workers : 4
[2023/10/06 13:21:14] ppocr INFO:         shuffle : False
[2023/10/06 13:21:14] ppocr INFO: Global : 
[2023/10/06 13:21:14] ppocr INFO:     cal_metric_during_train : True
[2023/10/06 13:21:14] ppocr INFO:     character_dict_path : ppocr/utils/dict/arabic_dict.txt
[2023/10/06 13:21:14] ppocr INFO:     checkpoints : None
[2023/10/06 13:21:14] ppocr INFO:     debug : False
[2023/10/06 13:21:14] ppocr INFO:     distributed : False
[2023/10/06 13:21:14] ppocr INFO:     epoch_num : 500
[2023/10/06 13:21:14] ppocr INFO:     eval_batch_step : [0, 2000]
[2023/10/06 13:21:14] ppocr INFO:     infer_img : ./doc/imgs_words/arabic/ar_2.jpg
[2023/10/06 13:21:14] ppocr INFO:     infer_mode : False
[2023/10/06 13:21:14] ppocr INFO:     log_smooth_window : 20
[2023/10/06 13:21:14] ppocr INFO:     max_text_length : 25
[2023/10/06 13:21:14] ppocr INFO:     pretrained_model : None
[2023/10/06 13:21:14] ppocr INFO:     print_batch_step : 10
[2023/10/06 13:21:14] ppocr INFO:     save_epoch_step : 3
[2023/10/06 13:21:14] ppocr INFO:     save_inference_dir : None
[2023/10/06 13:21:14] ppocr INFO:     save_model_dir : ./output/v3_arabic_mobile
[2023/10/06 13:21:14] ppocr INFO:     save_res_path : ./output/rec/predicts_ppocrv3_arabic.txt
[2023/10/06 13:21:14] ppocr INFO:     use_gpu : True
[2023/10/06 13:21:14] ppocr INFO:     use_space_char : True
[2023/10/06 13:21:14] ppocr INFO:     use_visualdl : False
[2023/10/06 13:21:14] ppocr INFO: Loss : 
[2023/10/06 13:21:14] ppocr INFO:     loss_config_list : 
[2023/10/06 13:21:14] ppocr INFO:         CTCLoss : None
[2023/10/06 13:21:14] ppocr INFO:         SARLoss : None
[2023/10/06 13:21:14] ppocr INFO:     name : MultiLoss
[2023/10/06 13:21:14] ppocr INFO: Metric : 
[2023/10/06 13:21:14] ppocr INFO:     ignore_space : False
[2023/10/06 13:21:14] ppocr INFO:     main_indicator : acc
[2023/10/06 13:21:14] ppocr INFO:     name : RecMetric
[2023/10/06 13:21:14] ppocr INFO: Optimizer : 
[2023/10/06 13:21:14] ppocr INFO:     beta1 : 0.9
[2023/10/06 13:21:14] ppocr INFO:     beta2 : 0.999
[2023/10/06 13:21:14] ppocr INFO:     lr : 
[2023/10/06 13:21:14] ppocr INFO:         learning_rate : 0.001
[2023/10/06 13:21:14] ppocr INFO:         name : Cosine
[2023/10/06 13:21:14] ppocr INFO:         warmup_epoch : 5
[2023/10/06 13:21:14] ppocr INFO:     name : Adam
[2023/10/06 13:21:14] ppocr INFO:     regularizer : 
[2023/10/06 13:21:14] ppocr INFO:         factor : 3e-05
[2023/10/06 13:21:14] ppocr INFO:         name : L2
[2023/10/06 13:21:14] ppocr INFO: PostProcess : 
[2023/10/06 13:21:14] ppocr INFO:     name : CTCLabelDecode
[2023/10/06 13:21:14] ppocr INFO: Train : 
[2023/10/06 13:21:14] ppocr INFO:     dataset : 
[2023/10/06 13:21:14] ppocr INFO:         data_dir : /additional_drive2/zain/dataset/raw_data/arabic_docs_combined_caparsoft_Sep22/train/
[2023/10/06 13:21:14] ppocr INFO:         ext_op_transform_idx : 1
[2023/10/06 13:21:14] ppocr INFO:         label_file_list : ['/additional_drive2/zain/dataset/raw_data/arabic_docs_combined_caparsoft_Sep22/train/paddle_rec_arr_updated.txt']
[2023/10/06 13:21:14] ppocr INFO:         name : SimpleDataSet
[2023/10/06 13:21:14] ppocr INFO:         transforms : 
[2023/10/06 13:21:14] ppocr INFO:             DecodeImage : 
[2023/10/06 13:21:14] ppocr INFO:                 channel_first : False
[2023/10/06 13:21:14] ppocr INFO:                 img_mode : BGR
[2023/10/06 13:21:14] ppocr INFO:             RecConAug : 
[2023/10/06 13:21:14] ppocr INFO:                 ext_data_num : 2
[2023/10/06 13:21:14] ppocr INFO:                 image_shape : [48, 320, 3]
[2023/10/06 13:21:14] ppocr INFO:                 prob : 0.5
[2023/10/06 13:21:14] ppocr INFO:             RecAug : None
[2023/10/06 13:21:14] ppocr INFO:             MultiLabelEncode : None
[2023/10/06 13:21:14] ppocr INFO:             RecResizeImg : 
[2023/10/06 13:21:14] ppocr INFO:                 image_shape : [3, 48, 320]
[2023/10/06 13:21:14] ppocr INFO:             KeepKeys : 
[2023/10/06 13:21:14] ppocr INFO:                 keep_keys : ['image', 'label_ctc', 'label_sar', 'length', 'valid_ratio']
[2023/10/06 13:21:14] ppocr INFO:     loader : 
[2023/10/06 13:21:14] ppocr INFO:         batch_size_per_card : 128
[2023/10/06 13:21:14] ppocr INFO:         drop_last : True
[2023/10/06 13:21:14] ppocr INFO:         num_workers : 4
[2023/10/06 13:21:14] ppocr INFO:         shuffle : True
[2023/10/06 13:21:14] ppocr INFO: profiler_options : None
[2023/10/06 13:21:14] ppocr INFO: train with paddle 2.5.1 and device Place(gpu:0)
[2023/10/06 13:21:14] ppocr INFO: Initialize indexs of datasets:['/additional_drive2/zain/dataset/raw_data/arabic_docs_combined_caparsoft_Sep22/train/paddle_rec_arr_updated.txt']
[2023/10/06 13:21:15] ppocr INFO: Initialize indexs of datasets:['/additional_drive2/zain/dataset/raw_data/arabic_docs_combined_caparsoft_Sep22/val/paddle_rec_arr_updated.txt']
[2023/10/06 13:21:16] ppocr INFO: train dataloader has 749 iters
[2023/10/06 13:21:16] ppocr INFO: valid dataloader has 82 iters
[2023/10/06 13:21:16] ppocr INFO: train from scratch
[2023/10/06 13:21:16] ppocr INFO: During the training process, after the 0th iteration, an evaluation is run every 2000 iterations
[2023/10/06 13:21:35] ppocr INFO: epoch: [1/500], global_step: 10, lr: 0.000001, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 1.08825 s, avg_batch_cost: 1.89337 s, avg_samples: 128.0, ips: 67.60450 samples/s, eta: 8 days, 4:57:26
[2023/10/06 13:21:48] ppocr INFO: epoch: [1/500], global_step: 20, lr: 0.000003, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.78203 s, avg_batch_cost: 1.33533 s, avg_samples: 128.0, ips: 95.85615 samples/s, eta: 6 days, 23:55:41
[2023/10/06 13:22:03] ppocr INFO: epoch: [1/500], global_step: 30, lr: 0.000005, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 1.00161 s, avg_batch_cost: 1.54638 s, avg_samples: 128.0, ips: 82.77376 samples/s, eta: 6 days, 21:34:01
[2023/10/06 13:22:15] ppocr INFO: epoch: [1/500], global_step: 40, lr: 0.000008, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.61786 s, avg_batch_cost: 1.16110 s, avg_samples: 128.0, ips: 110.24001 samples/s, eta: 6 days, 10:21:56
[2023/10/06 13:22:30] ppocr INFO: epoch: [1/500], global_step: 50, lr: 0.000011, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.89978 s, avg_batch_cost: 1.44257 s, avg_samples: 128.0, ips: 88.73038 samples/s, eta: 6 days, 9:29:55
[2023/10/06 13:22:40] ppocr INFO: epoch: [1/500], global_step: 60, lr: 0.000013, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.55627 s, avg_batch_cost: 1.09501 s, avg_samples: 128.0, ips: 116.89396 samples/s, eta: 6 days, 2:53:39
[2023/10/06 13:22:54] ppocr INFO: epoch: [1/500], global_step: 70, lr: 0.000016, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.79183 s, avg_batch_cost: 1.33457 s, avg_samples: 128.0, ips: 95.91138 samples/s, eta: 6 days, 1:44:07
[2023/10/06 13:23:05] ppocr INFO: epoch: [1/500], global_step: 80, lr: 0.000019, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.54361 s, avg_batch_cost: 1.07552 s, avg_samples: 128.0, ips: 119.01189 samples/s, eta: 5 days, 21:29:51
[2023/10/06 13:23:16] ppocr INFO: epoch: [1/500], global_step: 90, lr: 0.000021, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.59985 s, avg_batch_cost: 1.13322 s, avg_samples: 128.0, ips: 112.95207 samples/s, eta: 5 days, 18:52:03
[2023/10/06 13:23:27] ppocr INFO: epoch: [1/500], global_step: 100, lr: 0.000024, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.51949 s, avg_batch_cost: 1.06212 s, avg_samples: 128.0, ips: 120.51386 samples/s, eta: 5 days, 16:01:25
[2023/10/06 13:23:37] ppocr INFO: epoch: [1/500], global_step: 110, lr: 0.000027, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.52135 s, avg_batch_cost: 1.04407 s, avg_samples: 128.0, ips: 122.59768 samples/s, eta: 5 days, 13:31:31
[2023/10/06 13:23:48] ppocr INFO: epoch: [1/500], global_step: 120, lr: 0.000029, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.51163 s, avg_batch_cost: 1.07150 s, avg_samples: 128.0, ips: 119.45873 samples/s, eta: 5 days, 11:40:51
[2023/10/06 13:23:57] ppocr INFO: epoch: [1/500], global_step: 130, lr: 0.000032, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.37556 s, avg_batch_cost: 0.92996 s, avg_samples: 128.0, ips: 137.64060 samples/s, eta: 5 days, 8:59:14
[2023/10/06 13:24:07] ppocr INFO: epoch: [1/500], global_step: 140, lr: 0.000035, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.43143 s, avg_batch_cost: 0.95315 s, avg_samples: 128.0, ips: 134.29092 samples/s, eta: 5 days, 6:51:02
[2023/10/06 13:24:15] ppocr INFO: epoch: [1/500], global_step: 150, lr: 0.000037, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.28326 s, avg_batch_cost: 0.82487 s, avg_samples: 128.0, ips: 155.17676 samples/s, eta: 5 days, 4:06:32
[2023/10/06 13:24:25] ppocr INFO: epoch: [1/500], global_step: 160, lr: 0.000040, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.47801 s, avg_batch_cost: 1.01583 s, avg_samples: 128.0, ips: 126.00486 samples/s, eta: 5 days, 2:57:03
[2023/10/06 13:24:33] ppocr INFO: epoch: [1/500], global_step: 170, lr: 0.000043, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.25367 s, avg_batch_cost: 0.80017 s, avg_samples: 128.0, ips: 159.96559 samples/s, eta: 5 days, 0:36:35
[2023/10/06 13:24:41] ppocr INFO: epoch: [1/500], global_step: 180, lr: 0.000045, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.25914 s, avg_batch_cost: 0.81007 s, avg_samples: 128.0, ips: 158.01161 samples/s, eta: 4 days, 22:35:08
[2023/10/06 13:24:48] ppocr INFO: epoch: [1/500], global_step: 190, lr: 0.000048, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.17812 s, avg_batch_cost: 0.71288 s, avg_samples: 128.0, ips: 179.55362 samples/s, eta: 4 days, 20:14:32
[2023/10/06 13:24:56] ppocr INFO: epoch: [1/500], global_step: 200, lr: 0.000051, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.24905 s, avg_batch_cost: 0.77150 s, avg_samples: 128.0, ips: 165.91077 samples/s, eta: 4 days, 18:26:16
[2023/10/06 13:25:03] ppocr INFO: epoch: [1/500], global_step: 210, lr: 0.000053, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.15386 s, avg_batch_cost: 0.70094 s, avg_samples: 128.0, ips: 182.61269 samples/s, eta: 4 days, 16:27:21
[2023/10/06 13:25:10] ppocr INFO: epoch: [1/500], global_step: 220, lr: 0.000056, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.16276 s, avg_batch_cost: 0.71987 s, avg_samples: 128.0, ips: 177.80991 samples/s, eta: 4 days, 14:44:36
[2023/10/06 13:25:16] ppocr INFO: epoch: [1/500], global_step: 230, lr: 0.000059, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.07249 s, avg_batch_cost: 0.61327 s, avg_samples: 128.0, ips: 208.71682 samples/s, eta: 4 days, 12:41:51
[2023/10/06 13:25:23] ppocr INFO: epoch: [1/500], global_step: 240, lr: 0.000061, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.12566 s, avg_batch_cost: 0.67631 s, avg_samples: 128.0, ips: 189.26302 samples/s, eta: 4 days, 11:05:43
[2023/10/06 13:25:29] ppocr INFO: epoch: [1/500], global_step: 250, lr: 0.000064, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.04584 s, avg_batch_cost: 0.59374 s, avg_samples: 128.0, ips: 215.58130 samples/s, eta: 4 days, 9:16:40
[2023/10/06 13:25:34] ppocr INFO: epoch: [1/500], global_step: 260, lr: 0.000067, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00561 s, avg_batch_cost: 0.54258 s, avg_samples: 128.0, ips: 235.91162 samples/s, eta: 4 days, 7:23:43
[2023/10/06 13:25:40] ppocr INFO: epoch: [1/500], global_step: 270, lr: 0.000069, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.51401 s, avg_samples: 128.0, ips: 249.02132 samples/s, eta: 4 days, 5:32:32
[2023/10/06 13:25:45] ppocr INFO: epoch: [1/500], global_step: 280, lr: 0.000072, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.55473 s, avg_samples: 128.0, ips: 230.74335 samples/s, eta: 4 days, 3:58:21
[2023/10/06 13:25:50] ppocr INFO: epoch: [1/500], global_step: 290, lr: 0.000075, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.01990 s, avg_batch_cost: 0.54240 s, avg_samples: 128.0, ips: 235.98734 samples/s, eta: 4 days, 2:28:00
[2023/10/06 13:25:56] ppocr INFO: epoch: [1/500], global_step: 300, lr: 0.000077, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.54070 s, avg_samples: 128.0, ips: 236.72930 samples/s, eta: 4 days, 1:03:19
[2023/10/06 13:26:01] ppocr INFO: epoch: [1/500], global_step: 310, lr: 0.000080, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.53417 s, avg_samples: 128.0, ips: 239.62504 samples/s, eta: 3 days, 23:42:47
[2023/10/06 13:26:06] ppocr INFO: epoch: [1/500], global_step: 320, lr: 0.000083, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.52612 s, avg_samples: 128.0, ips: 243.28941 samples/s, eta: 3 days, 22:25:42
[2023/10/06 13:26:12] ppocr INFO: epoch: [1/500], global_step: 330, lr: 0.000085, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00016 s, avg_batch_cost: 0.53212 s, avg_samples: 128.0, ips: 240.54676 samples/s, eta: 3 days, 21:14:26
[2023/10/06 13:26:17] ppocr INFO: epoch: [1/500], global_step: 340, lr: 0.000088, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.52129 s, avg_samples: 128.0, ips: 245.54243 samples/s, eta: 3 days, 20:05:21
[2023/10/06 13:26:23] ppocr INFO: epoch: [1/500], global_step: 350, lr: 0.000091, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.55921 s, avg_samples: 128.0, ips: 228.89524 samples/s, eta: 3 days, 19:06:58
[2023/10/06 13:26:28] ppocr INFO: epoch: [1/500], global_step: 360, lr: 0.000093, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00016 s, avg_batch_cost: 0.54348 s, avg_samples: 128.0, ips: 235.52057 samples/s, eta: 3 days, 18:09:07
[2023/10/06 13:26:33] ppocr INFO: epoch: [1/500], global_step: 370, lr: 0.000096, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.52004 s, avg_samples: 128.0, ips: 246.13627 samples/s, eta: 3 days, 17:10:25
[2023/10/06 13:26:39] ppocr INFO: epoch: [1/500], global_step: 380, lr: 0.000099, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.53828 s, avg_samples: 128.0, ips: 237.79401 samples/s, eta: 3 days, 16:17:48
[2023/10/06 13:26:44] ppocr INFO: epoch: [1/500], global_step: 390, lr: 0.000101, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.52969 s, avg_samples: 128.0, ips: 241.64856 samples/s, eta: 3 days, 15:26:31
[2023/10/06 13:26:49] ppocr INFO: epoch: [1/500], global_step: 400, lr: 0.000104, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.53065 s, avg_samples: 128.0, ips: 241.21462 samples/s, eta: 3 days, 14:37:55
[2023/10/06 13:26:55] ppocr INFO: epoch: [1/500], global_step: 410, lr: 0.000107, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.53041 s, avg_samples: 128.0, ips: 241.32191 samples/s, eta: 3 days, 13:51:40
[2023/10/06 13:27:00] ppocr INFO: epoch: [1/500], global_step: 420, lr: 0.000109, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.54004 s, avg_samples: 128.0, ips: 237.02089 samples/s, eta: 3 days, 13:09:02
[2023/10/06 13:27:05] ppocr INFO: epoch: [1/500], global_step: 430, lr: 0.000112, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.52477 s, avg_samples: 128.0, ips: 243.91501 samples/s, eta: 3 days, 12:26:11
[2023/10/06 13:27:10] ppocr INFO: epoch: [1/500], global_step: 440, lr: 0.000115, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.52309 s, avg_samples: 128.0, ips: 244.70179 samples/s, eta: 3 days, 11:45:01
[2023/10/06 13:27:16] ppocr INFO: epoch: [1/500], global_step: 450, lr: 0.000117, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.52461 s, avg_samples: 128.0, ips: 243.99003 samples/s, eta: 3 days, 11:05:54
[2023/10/06 13:27:21] ppocr INFO: epoch: [1/500], global_step: 460, lr: 0.000120, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.55077 s, avg_samples: 128.0, ips: 232.40186 samples/s, eta: 3 days, 10:32:01
[2023/10/06 13:27:27] ppocr INFO: epoch: [1/500], global_step: 470, lr: 0.000123, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.53369 s, avg_samples: 128.0, ips: 239.83958 samples/s, eta: 3 days, 9:57:19
[2023/10/06 13:27:32] ppocr INFO: epoch: [1/500], global_step: 480, lr: 0.000125, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.54482 s, avg_samples: 128.0, ips: 234.93821 samples/s, eta: 3 days, 9:25:30
[2023/10/06 13:27:37] ppocr INFO: epoch: [1/500], global_step: 490, lr: 0.000128, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.52896 s, avg_samples: 128.0, ips: 241.98328 samples/s, eta: 3 days, 8:52:57
[2023/10/06 13:27:43] ppocr INFO: epoch: [1/500], global_step: 500, lr: 0.000131, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.52320 s, avg_samples: 128.0, ips: 244.64700 samples/s, eta: 3 days, 8:21:00
[2023/10/06 13:27:48] ppocr INFO: epoch: [1/500], global_step: 510, lr: 0.000133, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.51550 s, avg_samples: 128.0, ips: 248.30066 samples/s, eta: 3 days, 7:49:21
[2023/10/06 13:27:53] ppocr INFO: epoch: [1/500], global_step: 520, lr: 0.000136, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.52065 s, avg_samples: 128.0, ips: 245.84786 samples/s, eta: 3 days, 7:19:31
[2023/10/06 13:27:58] ppocr INFO: epoch: [1/500], global_step: 530, lr: 0.000139, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.52368 s, avg_samples: 128.0, ips: 244.42396 samples/s, eta: 3 days, 6:51:11
[2023/10/06 13:28:03] ppocr INFO: epoch: [1/500], global_step: 540, lr: 0.000141, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.53489 s, avg_samples: 128.0, ips: 239.30113 samples/s, eta: 3 days, 6:25:11
[2023/10/06 13:28:09] ppocr INFO: epoch: [1/500], global_step: 550, lr: 0.000144, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.53439 s, avg_samples: 128.0, ips: 239.52446 samples/s, eta: 3 days, 6:00:04
[2023/10/06 13:28:14] ppocr INFO: epoch: [1/500], global_step: 560, lr: 0.000147, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.53297 s, avg_samples: 128.0, ips: 240.16332 samples/s, eta: 3 days, 5:35:41
[2023/10/06 13:28:20] ppocr INFO: epoch: [1/500], global_step: 570, lr: 0.000149, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.53791 s, avg_samples: 128.0, ips: 237.95709 samples/s, eta: 3 days, 5:12:42
[2023/10/06 13:28:25] ppocr INFO: epoch: [1/500], global_step: 580, lr: 0.000152, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.54265 s, avg_samples: 128.0, ips: 235.88022 samples/s, eta: 3 days, 4:51:00
[2023/10/06 13:28:30] ppocr INFO: epoch: [1/500], global_step: 590, lr: 0.000155, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.53701 s, avg_samples: 128.0, ips: 238.35654 samples/s, eta: 3 days, 4:29:27
[2023/10/06 13:28:36] ppocr INFO: epoch: [1/500], global_step: 600, lr: 0.000157, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.56142 s, avg_samples: 128.0, ips: 227.99258 samples/s, eta: 3 days, 4:11:09
[2023/10/06 13:28:41] ppocr INFO: epoch: [1/500], global_step: 610, lr: 0.000160, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.54799 s, avg_samples: 128.0, ips: 233.58242 samples/s, eta: 3 days, 3:52:04
[2023/10/06 13:28:47] ppocr INFO: epoch: [1/500], global_step: 620, lr: 0.000163, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.53408 s, avg_samples: 128.0, ips: 239.66634 samples/s, eta: 3 days, 3:32:13
[2023/10/06 13:28:52] ppocr INFO: epoch: [1/500], global_step: 630, lr: 0.000165, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.56540 s, avg_samples: 128.0, ips: 226.38746 samples/s, eta: 3 days, 3:16:04
[2023/10/06 13:28:58] ppocr INFO: epoch: [1/500], global_step: 640, lr: 0.000168, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.56888 s, avg_samples: 128.0, ips: 225.00174 samples/s, eta: 3 days, 3:00:47
[2023/10/06 13:29:04] ppocr INFO: epoch: [1/500], global_step: 650, lr: 0.000171, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.56244 s, avg_samples: 128.0, ips: 227.58003 samples/s, eta: 3 days, 2:45:20
[2023/10/06 13:29:09] ppocr INFO: epoch: [1/500], global_step: 660, lr: 0.000173, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.53802 s, avg_samples: 128.0, ips: 237.90886 samples/s, eta: 3 days, 2:28:03
[2023/10/06 13:29:15] ppocr INFO: epoch: [1/500], global_step: 670, lr: 0.000176, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.54924 s, avg_samples: 128.0, ips: 233.04895 samples/s, eta: 3 days, 2:12:19
[2023/10/06 13:29:20] ppocr INFO: epoch: [1/500], global_step: 680, lr: 0.000179, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.54630 s, avg_samples: 128.0, ips: 234.30499 samples/s, eta: 3 days, 1:56:47
[2023/10/06 13:29:25] ppocr INFO: epoch: [1/500], global_step: 690, lr: 0.000181, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.52432 s, avg_samples: 128.0, ips: 244.12569 samples/s, eta: 3 days, 1:39:42
[2023/10/06 13:29:31] ppocr INFO: epoch: [1/500], global_step: 700, lr: 0.000184, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.53362 s, avg_samples: 128.0, ips: 239.86984 samples/s, eta: 3 days, 1:23:56
[2023/10/06 13:29:36] ppocr INFO: epoch: [1/500], global_step: 710, lr: 0.000187, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.53125 s, avg_samples: 128.0, ips: 240.94088 samples/s, eta: 3 days, 1:08:24
[2023/10/06 13:29:41] ppocr INFO: epoch: [1/500], global_step: 720, lr: 0.000189, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.52896 s, avg_samples: 128.0, ips: 241.98269 samples/s, eta: 3 days, 0:53:07
[2023/10/06 13:29:47] ppocr INFO: epoch: [1/500], global_step: 730, lr: 0.000192, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.52500 s, avg_samples: 128.0, ips: 243.80807 samples/s, eta: 3 days, 0:37:53
[2023/10/06 13:29:52] ppocr INFO: epoch: [1/500], global_step: 740, lr: 0.000195, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.54336 s, avg_samples: 128.0, ips: 235.57141 samples/s, eta: 3 days, 0:24:37
[2023/10/06 13:29:57] ppocr INFO: epoch: [1/500], global_step: 749, lr: 0.000197, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00024 s, avg_batch_cost: 0.48613 s, avg_samples: 115.2, ips: 236.97573 samples/s, eta: 3 days, 0:12:45
[2023/10/06 13:29:57] ppocr INFO: save model in ./output/v3_arabic_mobile/latest
[2023/10/06 13:29:58] ppocr INFO: epoch: [2/500], global_step: 750, lr: 0.000197, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.08468 s, avg_batch_cost: 0.14658 s, avg_samples: 12.8, ips: 87.32420 samples/s, eta: 3 days, 0:19:08
[2023/10/06 13:30:04] ppocr INFO: epoch: [2/500], global_step: 760, lr: 0.000200, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.54022 s, avg_samples: 128.0, ips: 236.94009 samples/s, eta: 3 days, 0:06:12
[2023/10/06 13:30:09] ppocr INFO: epoch: [2/500], global_step: 770, lr: 0.000203, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.54077 s, avg_samples: 128.0, ips: 236.69792 samples/s, eta: 2 days, 23:53:39
[2023/10/06 13:30:14] ppocr INFO: epoch: [2/500], global_step: 780, lr: 0.000205, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.52389 s, avg_samples: 128.0, ips: 244.32843 samples/s, eta: 2 days, 23:40:04
[2023/10/06 13:30:20] ppocr INFO: epoch: [2/500], global_step: 790, lr: 0.000208, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.53729 s, avg_samples: 128.0, ips: 238.23244 samples/s, eta: 2 days, 23:27:53
[2023/10/06 13:30:25] ppocr INFO: epoch: [2/500], global_step: 800, lr: 0.000211, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.52781 s, avg_samples: 128.0, ips: 242.50922 samples/s, eta: 2 days, 23:15:16
[2023/10/06 13:30:30] ppocr INFO: epoch: [2/500], global_step: 810, lr: 0.000213, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.53596 s, avg_samples: 128.0, ips: 238.82419 samples/s, eta: 2 days, 23:03:35
[2023/10/06 13:30:36] ppocr INFO: epoch: [2/500], global_step: 820, lr: 0.000216, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.52174 s, avg_samples: 128.0, ips: 245.33112 samples/s, eta: 2 days, 22:51:06
[2023/10/06 13:30:41] ppocr INFO: epoch: [2/500], global_step: 830, lr: 0.000219, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.53321 s, avg_samples: 128.0, ips: 240.05770 samples/s, eta: 2 days, 22:39:46
[2023/10/06 13:30:46] ppocr INFO: epoch: [2/500], global_step: 840, lr: 0.000221, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.51663 s, avg_samples: 128.0, ips: 247.75835 samples/s, eta: 2 days, 22:27:29
[2023/10/06 13:30:51] ppocr INFO: epoch: [2/500], global_step: 850, lr: 0.000224, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.52712 s, avg_samples: 128.0, ips: 242.82905 samples/s, eta: 2 days, 22:16:16
[2023/10/06 13:30:57] ppocr INFO: epoch: [2/500], global_step: 860, lr: 0.000227, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.53367 s, avg_samples: 128.0, ips: 239.84981 samples/s, eta: 2 days, 22:05:46
[2023/10/06 13:31:02] ppocr INFO: epoch: [2/500], global_step: 870, lr: 0.000230, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.52799 s, avg_samples: 128.0, ips: 242.42656 samples/s, eta: 2 days, 21:55:06
[2023/10/06 13:31:07] ppocr INFO: epoch: [2/500], global_step: 880, lr: 0.000232, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.52588 s, avg_samples: 128.0, ips: 243.40093 samples/s, eta: 2 days, 21:44:32
[2023/10/06 13:31:13] ppocr INFO: epoch: [2/500], global_step: 890, lr: 0.000235, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.54086 s, avg_samples: 128.0, ips: 236.66026 samples/s, eta: 2 days, 21:35:15
[2023/10/06 13:31:18] ppocr INFO: epoch: [2/500], global_step: 900, lr: 0.000238, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.54240 s, avg_samples: 128.0, ips: 235.98816 samples/s, eta: 2 days, 21:26:16
[2023/10/06 13:31:23] ppocr INFO: epoch: [2/500], global_step: 910, lr: 0.000240, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.53966 s, avg_samples: 128.0, ips: 237.18535 samples/s, eta: 2 days, 21:17:18
[2023/10/06 13:31:29] ppocr INFO: epoch: [2/500], global_step: 920, lr: 0.000243, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.51186 s, avg_samples: 128.0, ips: 250.06955 samples/s, eta: 2 days, 21:06:39
[2023/10/06 13:31:34] ppocr INFO: epoch: [2/500], global_step: 930, lr: 0.000246, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.51984 s, avg_samples: 128.0, ips: 246.22771 samples/s, eta: 2 days, 20:56:45
[2023/10/06 13:31:39] ppocr INFO: epoch: [2/500], global_step: 940, lr: 0.000248, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.54609 s, avg_samples: 128.0, ips: 234.39172 samples/s, eta: 2 days, 20:48:48
[2023/10/06 13:31:45] ppocr INFO: epoch: [2/500], global_step: 950, lr: 0.000251, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.54968 s, avg_samples: 128.0, ips: 232.86372 samples/s, eta: 2 days, 20:41:15
[2023/10/06 13:31:50] ppocr INFO: epoch: [2/500], global_step: 960, lr: 0.000254, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.52278 s, avg_samples: 128.0, ips: 244.84689 samples/s, eta: 2 days, 20:32:07
[2023/10/06 13:31:55] ppocr INFO: epoch: [2/500], global_step: 970, lr: 0.000256, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.54493 s, avg_samples: 128.0, ips: 234.89217 samples/s, eta: 2 days, 20:24:36
[2023/10/06 13:32:01] ppocr INFO: epoch: [2/500], global_step: 980, lr: 0.000259, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.54489 s, avg_samples: 128.0, ips: 234.90906 samples/s, eta: 2 days, 20:17:13
[2023/10/06 13:32:06] ppocr INFO: epoch: [2/500], global_step: 990, lr: 0.000262, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.53534 s, avg_samples: 128.0, ips: 239.10099 samples/s, eta: 2 days, 20:09:23
[2023/10/06 13:32:12] ppocr INFO: epoch: [2/500], global_step: 1000, lr: 0.000264, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.53432 s, avg_samples: 128.0, ips: 239.55743 samples/s, eta: 2 days, 20:01:38
[2023/10/06 13:32:17] ppocr INFO: epoch: [2/500], global_step: 1010, lr: 0.000267, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.56337 s, avg_samples: 128.0, ips: 227.20421 samples/s, eta: 2 days, 19:55:50
[2023/10/06 13:32:23] ppocr INFO: epoch: [2/500], global_step: 1020, lr: 0.000270, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.54073 s, avg_samples: 128.0, ips: 236.71596 samples/s, eta: 2 days, 19:48:46
[2023/10/06 13:32:28] ppocr INFO: epoch: [2/500], global_step: 1030, lr: 0.000272, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.53771 s, avg_samples: 128.0, ips: 238.04639 samples/s, eta: 2 days, 19:41:39
[2023/10/06 13:32:34] ppocr INFO: epoch: [2/500], global_step: 1040, lr: 0.000275, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.55312 s, avg_samples: 128.0, ips: 231.41354 samples/s, eta: 2 days, 19:35:36
[2023/10/06 13:32:39] ppocr INFO: epoch: [2/500], global_step: 1050, lr: 0.000278, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.53664 s, avg_samples: 128.0, ips: 238.52127 samples/s, eta: 2 days, 19:28:41
[2023/10/06 13:32:44] ppocr INFO: epoch: [2/500], global_step: 1060, lr: 0.000280, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.51724 s, avg_samples: 128.0, ips: 247.46538 samples/s, eta: 2 days, 19:20:45
[2023/10/06 13:32:49] ppocr INFO: epoch: [2/500], global_step: 1070, lr: 0.000283, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.53816 s, avg_samples: 128.0, ips: 237.84742 samples/s, eta: 2 days, 19:14:11
[2023/10/06 13:32:55] ppocr INFO: epoch: [2/500], global_step: 1080, lr: 0.000286, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.53815 s, avg_samples: 128.0, ips: 237.85251 samples/s, eta: 2 days, 19:07:44
[2023/10/06 13:33:00] ppocr INFO: epoch: [2/500], global_step: 1090, lr: 0.000288, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.54463 s, avg_samples: 128.0, ips: 235.01980 samples/s, eta: 2 days, 19:01:46
[2023/10/06 13:33:06] ppocr INFO: epoch: [2/500], global_step: 1100, lr: 0.000291, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.54392 s, avg_samples: 128.0, ips: 235.32664 samples/s, eta: 2 days, 18:55:52
[2023/10/06 13:33:11] ppocr INFO: epoch: [2/500], global_step: 1110, lr: 0.000294, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.54504 s, avg_samples: 128.0, ips: 234.84665 samples/s, eta: 2 days, 18:50:09
[2023/10/06 13:33:17] ppocr INFO: epoch: [2/500], global_step: 1120, lr: 0.000296, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.56626 s, avg_samples: 128.0, ips: 226.04642 samples/s, eta: 2 days, 18:45:42
[2023/10/06 13:33:22] ppocr INFO: epoch: [2/500], global_step: 1130, lr: 0.000299, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.54136 s, avg_samples: 128.0, ips: 236.44196 samples/s, eta: 2 days, 18:39:57
[2023/10/06 13:33:28] ppocr INFO: epoch: [2/500], global_step: 1140, lr: 0.000302, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.54374 s, avg_samples: 128.0, ips: 235.40616 samples/s, eta: 2 days, 18:34:26
[2023/10/06 13:33:33] ppocr INFO: epoch: [2/500], global_step: 1150, lr: 0.000304, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.54877 s, avg_samples: 128.0, ips: 233.25011 samples/s, eta: 2 days, 18:29:17
[2023/10/06 13:33:39] ppocr INFO: epoch: [2/500], global_step: 1160, lr: 0.000307, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.54821 s, avg_samples: 128.0, ips: 233.48869 samples/s, eta: 2 days, 18:24:12
[2023/10/06 13:33:44] ppocr INFO: epoch: [2/500], global_step: 1170, lr: 0.000310, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.55393 s, avg_samples: 128.0, ips: 231.07563 samples/s, eta: 2 days, 18:19:30
[2023/10/06 13:33:50] ppocr INFO: epoch: [2/500], global_step: 1180, lr: 0.000312, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.53854 s, avg_samples: 128.0, ips: 237.68089 samples/s, eta: 2 days, 18:14:04
.
.
.
[2023/10/06 14:56:57] ppocr INFO: epoch: [14/500], global_step: 10440, lr: 0.000999, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.47577 s, avg_samples: 128.0, ips: 269.03580 samples/s, eta: 2 days, 7:16:19

I'm feeling a bit confused, and I wonder if I might be doing something incorrectly. I've been looking into several GitHub issues related to PaddleOCR, and most of them seem to be about achieving an accuracy of 0. The contributors to PaddleOCR have suggested that increasing the number of epochs can resolve this issue. I've observed many of these discussions, and they all share a common pattern. The loss starts decreasing from the very first epoch, and there are no instances of loss being "nanxxx." But in my case I am getting loss: nanxxx from start of epoch. I train it for 1500 epoches and it take 3 days, but didn't found any change in the loss and accuracy.

I'm seeking assistance and advice from anyone who might be able to shed light on this matter.

Oct 06 '23 10:10 IbrarBabar009

Use https://github.com/PaddlePaddle/PaddleOCR/tree/release/2.6.1 this version of paddleOCR instead of cloning current version I also find some errors while using the current version, download the link I provided as zip. And do the same thing and it will work, if you want to install paddleocr in venv, use this paddleocr==2.6.1.3 use this version. It will definitely work.

Oct 06 '23 10:10 hritikakolkar

This time it will work wait for 4 hours the model till you see results as you are finetuning.

Oct 06 '23 10:10 hritikakolkar

thank you so much for response, will try it and update you soon. and python version you used?

Oct 06 '23 11:10 IbrarBabar009

@hritikakolkar just download and unzip and use paddleOCR-2.6.1 version

and follow the same steps I didn't use paddleocr==2.6.1.3, as this version might be for CPU training and I am on GPU. I have install latest version of Paddleocr. pip install paddlepaddle-gpu. and this is the latest version I am using 2.5.1

Although I am not finetuning write now, I am just training, but same issue when I finetune that model.

Lastly I am confused about the ppocr/utils/dict/arabic_dict.txt should it needs to be updated?

and start training paddleocr and here are the results yet (issue not resolved yet), let's wait for some epoches and then i will let you know about the update

(PaddleOCR1) root@gpu1:/additional_drive/ibrar/6/PaddleOCR# python3 tools/train.py -c configs/rec/PP-OCRv3/multi_language/arabic_PP-OCRv3_rec.yml
[2023/10/06 17:21:17] ppocr INFO: Architecture :
[2023/10/06 17:21:17] ppocr INFO:     Backbone :
[2023/10/06 17:21:17] ppocr INFO:         last_conv_stride : [1, 2]
[2023/10/06 17:21:17] ppocr INFO:         last_pool_type : avg
[2023/10/06 17:21:17] ppocr INFO:         name : MobileNetV1Enhance
[2023/10/06 17:21:17] ppocr INFO:         scale : 0.5
[2023/10/06 17:21:17] ppocr INFO:     Head :
[2023/10/06 17:21:17] ppocr INFO:         head_list :
[2023/10/06 17:21:17] ppocr INFO:             CTCHead :
[2023/10/06 17:21:17] ppocr INFO:                 Head :
[2023/10/06 17:21:17] ppocr INFO:                     fc_decay : 1e-05
[2023/10/06 17:21:17] ppocr INFO:                 Neck :
[2023/10/06 17:21:17] ppocr INFO:                     depth : 2
[2023/10/06 17:21:17] ppocr INFO:                     dims : 64
[2023/10/06 17:21:17] ppocr INFO:                     hidden_dims : 120
[2023/10/06 17:21:17] ppocr INFO:                     name : svtr
[2023/10/06 17:21:17] ppocr INFO:                     use_guide : True
[2023/10/06 17:21:17] ppocr INFO:             SARHead :
[2023/10/06 17:21:17] ppocr INFO:                 enc_dim : 512
[2023/10/06 17:21:17] ppocr INFO:                 max_text_length : 25
[2023/10/06 17:21:17] ppocr INFO:         name : MultiHead
[2023/10/06 17:21:17] ppocr INFO:     Transform : None
[2023/10/06 17:21:17] ppocr INFO:     algorithm : SVTR_LCNet
[2023/10/06 17:21:17] ppocr INFO:     model_type : rec
[2023/10/06 17:21:17] ppocr INFO: Eval :
[2023/10/06 17:21:17] ppocr INFO:     dataset :
[2023/10/06 17:21:17] ppocr INFO:         data_dir : /additional_drive2/zain/dataset/raw_data/arabic_docs_combined_caparsoft_Sep22/val/
[2023/10/06 17:21:17] ppocr INFO:         label_file_list : ['/additional_drive2/zain/dataset/raw_data/arabic_docs_combined_caparsoft_Sep22/val/paddle_rec_arr_updated.txt']
[2023/10/06 17:21:17] ppocr INFO:         name : SimpleDataSet
[2023/10/06 17:21:17] ppocr INFO:         transforms :
[2023/10/06 17:21:17] ppocr INFO:             DecodeImage :[2023/10/06 17:21:17] ppocr INFO:                 channel_first : False
[2023/10/06 17:21:17] ppocr INFO:                 img_mode : BGR
[2023/10/06 17:21:17] ppocr INFO:             MultiLabelEncode : None
[2023/10/06 17:21:17] ppocr INFO:             RecResizeImg :
[2023/10/06 17:21:17] ppocr INFO:                 image_shape : [3, 48, 320]
[2023/10/06 17:21:17] ppocr INFO:             KeepKeys :
[2023/10/06 17:21:17] ppocr INFO:                 keep_keys : ['image', 'label_ctc', 'label_sar', 'length', 'valid_ratio']
[2023/10/06 17:21:17] ppocr INFO:     loader :
[2023/10/06 17:21:17] ppocr INFO:         batch_size_per_card : 128
[2023/10/06 17:21:17] ppocr INFO:         drop_last : False
[2023/10/06 17:21:17] ppocr INFO:         num_workers : 4
[2023/10/06 17:21:17] ppocr INFO:         shuffle : False
[2023/10/06 17:21:17] ppocr INFO: Global :
[2023/10/06 17:21:17] ppocr INFO:     cal_metric_during_train : True
[2023/10/06 17:21:17] ppocr INFO:     character_dict_path : ppocr/utils/dict/arabic_dict.txt
[2023/10/06 17:21:17] ppocr INFO:     checkpoints : None
[2023/10/06 17:21:17] ppocr INFO:     debug : False
[2023/10/06 17:21:17] ppocr INFO:     distributed : False
[2023/10/06 17:21:17] ppocr INFO:     epoch_num : 500
[2023/10/06 17:21:17] ppocr INFO:     eval_batch_step : [0, 2000]
[2023/10/06 17:21:17] ppocr INFO:     infer_img : ./doc/imgs_words/arabic/ar_2.jpg
[2023/10/06 17:21:17] ppocr INFO:     infer_mode : False
[2023/10/06 17:21:17] ppocr INFO:     log_smooth_window : 20
[2023/10/06 17:21:17] ppocr INFO:     max_text_length : 25
[2023/10/06 17:21:17] ppocr INFO:     pretrained_model : None
[2023/10/06 17:21:17] ppocr INFO:     print_batch_step : 10
[2023/10/06 17:21:17] ppocr INFO:     save_epoch_step : 3
[2023/10/06 17:21:17] ppocr INFO:     save_inference_dir : None
[2023/10/06 17:21:17] ppocr INFO:     save_model_dir : ./output/v3_arabic_mobile
[2023/10/06 17:21:17] ppocr INFO:     save_res_path : ./output/rec/predicts_ppocrv3_arabic.txt
[2023/10/06 17:21:17] ppocr INFO:     use_gpu : True
[2023/10/06 17:21:17] ppocr INFO:     use_space_char : True
[2023/10/06 17:21:17] ppocr INFO:     use_visualdl : False
[2023/10/06 17:21:17] ppocr INFO: Loss :
[2023/10/06 17:21:17] ppocr INFO:     loss_config_list :
[2023/10/06 17:21:17] ppocr INFO:         CTCLoss : None
[2023/10/06 17:21:17] ppocr INFO:         SARLoss : None
[2023/10/06 17:21:17] ppocr INFO:     name : MultiLoss
[2023/10/06 17:21:17] ppocr INFO: Metric :
[2023/10/06 17:21:17] ppocr INFO:     ignore_space : False
[2023/10/06 17:21:17] ppocr INFO:     main_indicator : acc
[2023/10/06 17:21:17] ppocr INFO:     name : RecMetric
[2023/10/06 17:21:17] ppocr INFO: Optimizer :
[2023/10/06 17:21:17] ppocr INFO:     beta1 : 0.9
[2023/10/06 17:21:17] ppocr INFO:     beta2 : 0.999
[2023/10/06 17:21:17] ppocr INFO:     lr :
[2023/10/06 17:21:17] ppocr INFO:         learning_rate : 0.001
[2023/10/06 17:21:17] ppocr INFO:         name : Cosine
[2023/10/06 17:21:17] ppocr INFO:         warmup_epoch : 5
[2023/10/06 17:21:17] ppocr INFO:     name : Adam
[2023/10/06 17:21:17] ppocr INFO:     regularizer :
[2023/10/06 17:21:17] ppocr INFO:         factor : 3e-05
[2023/10/06 17:21:17] ppocr INFO:         name : L2
[2023/10/06 17:21:17] ppocr INFO: PostProcess :
[2023/10/06 17:21:17] ppocr INFO:     name : CTCLabelDecode
[2023/10/06 17:21:17] ppocr INFO: Train :
[2023/10/06 17:21:17] ppocr INFO:     dataset :
[2023/10/06 17:21:17] ppocr INFO:         data_dir : /additional_drive2/zain/dataset/raw_data/arabic_docs_combined_caparsoft_Sep22/train/
[2023/10/06 17:21:17] ppocr INFO:         ext_op_transform_idx : 1
[2023/10/06 17:21:17] ppocr INFO:         label_file_list : ['/additional_drive2/zain/dataset/raw_data/arabic_docs_combined_caparsoft_Sep22/train/paddle_rec_arr_updated.txt']
[2023/10/06 17:21:17] ppocr INFO:         name : SimpleDataSet
[2023/10/06 17:21:17] ppocr INFO:         transforms :
[2023/10/06 17:21:17] ppocr INFO:             DecodeImage :
[2023/10/06 17:21:17] ppocr INFO:                 channel_first : False
[2023/10/06 17:21:17] ppocr INFO:                 img_mode : BGR
[2023/10/06 17:21:17] ppocr INFO:             RecConAug :
[2023/10/06 17:21:17] ppocr INFO:                 ext_data_num : 2
[2023/10/06 17:21:17] ppocr INFO:                 image_shape : [48, 320, 3]
[2023/10/06 17:21:17] ppocr INFO:                 prob : 0.5
[2023/10/06 17:21:17] ppocr INFO:             RecAug : None
[2023/10/06 17:21:17] ppocr INFO:             MultiLabelEncode : None
[2023/10/06 17:21:17] ppocr INFO:             RecResizeImg :
[2023/10/06 17:21:17] ppocr INFO:                 image_shape : [3, 48, 320]
[2023/10/06 17:21:17] ppocr INFO:             KeepKeys :
[2023/10/06 17:21:17] ppocr INFO:                 keep_keys : ['image', 'label_ctc', 'label_sar', 'length', 'valid_ratio']
[2023/10/06 17:21:17] ppocr INFO:     loader :
[2023/10/06 17:21:17] ppocr INFO:         batch_size_per_card : 128
[2023/10/06 17:21:17] ppocr INFO:         drop_last : True
[2023/10/06 17:21:17] ppocr INFO:         num_workers : 4
[2023/10/06 17:21:17] ppocr INFO:         shuffle : True
[2023/10/06 17:21:17] ppocr INFO: profiler_options : None
[2023/10/06 17:21:17] ppocr INFO: train with paddle 2.5.1 and device Place(gpu:0)
[2023/10/06 17:21:17] ppocr INFO: Initialize indexs of datasets:['/additional_drive2/zain/dataset/raw_data/arabic_docs_combined_caparsoft_Sep22/train/paddle_rec_arr_updated.txt']
'MakeBorderMap'
[2023/10/06 17:21:17] ppocr INFO: Initialize indexs of datasets:['/additional_drive2/zain/dataset/raw_data/arabic_docs_combined_caparsoft_Sep22/val/paddle_rec_arr_updated.txt']
W1006 17:21:17.844991 1990986 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 8.6, Driver API Version: 12.1, Runtime API Version: 11.8
W1006 17:21:17.846884 1990986 gpu_resources.cc:149] device: 0, cuDNN Version: 7.6.
[2023/10/06 17:21:18] ppocr INFO: train dataloader has 749 iters
[2023/10/06 17:21:18] ppocr INFO: valid dataloader has 82 iters
[2023/10/06 17:21:18] ppocr INFO: train from scratch
[2023/10/06 17:21:18] ppocr INFO: During the training process, after the 0th iteration, an evaluation is run every 2000 iterations
W1006 17:21:22.686206 1990986 gpu_resources.cc:275] WARNING: device: . The installed Paddle is compiled with CUDNN 8.6, but CUDNN version in your machine is 7.6, which may cause serious incompatible bug. Please recompile or reinstall Paddle withcompatible CUDNN version.
[2023/10/06 17:21:30] ppocr INFO: epoch: [1/500], global_step: 10, lr: 0.000001, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.66523 s, avg_batch_cost: 1.19455 s, avg_samples: 128.0, ips: 107.15345 samples/s, eta: 5 days, 4:15:46
[2023/10/06 17:21:39] ppocr INFO: epoch: [1/500], global_step: 20, lr: 0.000003, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.61352 s, avg_batch_cost: 0.92008 s, avg_samples: 128.0, ips: 139.11797 samples/s, eta: 4 days, 13:59:03
[2023/10/06 17:21:48] ppocr INFO: epoch: [1/500], global_step: 30, lr: 0.000005, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.59251 s, avg_batch_cost: 0.89967 s, avg_samples: 128.0, ips: 142.27390 samples/s, eta: 4 days, 8:30:55
[2023/10/06 17:21:57] ppocr INFO: epoch: [1/500], global_step: 40, lr: 0.000008, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.52084 s, avg_batch_cost: 0.82055 s, avg_samples: 128.0, ips: 155.99294 samples/s, eta: 4 days, 3:43:19
[2023/10/06 17:22:05] ppocr INFO: epoch: [1/500], global_step: 50, lr: 0.000011, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.57723 s, avg_batch_cost: 0.88061 s, avg_samples: 128.0, ips: 145.35325 samples/s, eta: 4 days, 2:05:41
[2023/10/06 17:22:12] ppocr INFO: epoch: [1/500], global_step: 60, lr: 0.000013, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.38603 s, avg_batch_cost: 0.68977 s, avg_samples: 128.0, ips: 185.56818 samples/s, eta: 3 days, 21:42:03
[2023/10/06 17:22:21] ppocr INFO: epoch: [1/500], global_step: 70, lr: 0.000016, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.57900 s, avg_batch_cost: 0.88394 s, avg_samples: 128.0, ips: 144.80584 samples/s, eta: 3 days, 21:26:48
[2023/10/06 17:22:27] ppocr INFO: epoch: [1/500], global_step: 80, lr: 0.000019, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.31529 s, avg_batch_cost: 0.61529 s, avg_samples: 128.0, ips: 208.03234 samples/s, eta: 3 days, 17:45:46
[2023/10/06 17:22:35] ppocr INFO: epoch: [1/500], global_step: 90, lr: 0.000021, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.49823 s, avg_batch_cost: 0.79881 s, avg_samples: 128.0, ips: 160.23821 samples/s, eta: 3 days, 17:01:05
[2023/10/06 17:22:42] ppocr INFO: epoch: [1/500], global_step: 100, lr: 0.000024, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.36666 s, avg_batch_cost: 0.66873 s, avg_samples: 128.0, ips: 191.40873 samples/s, eta: 3 days, 15:04:08
[2023/10/06 17:22:49] ppocr INFO: epoch: [1/500], global_step: 110, lr: 0.000027, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.45102 s, avg_batch_cost: 0.75185 s, avg_samples: 128.0, ips: 170.24643 samples/s, eta: 3 days, 14:15:34
[2023/10/06 17:22:56] ppocr INFO: epoch: [1/500], global_step: 120, lr: 0.000029, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.37003 s, avg_batch_cost: 0.66999 s, avg_samples: 128.0, ips: 191.04824 samples/s, eta: 3 days, 12:52:31
[2023/10/06 17:23:02] ppocr INFO: epoch: [1/500], global_step: 130, lr: 0.000032, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.30084 s, avg_batch_cost: 0.60359 s, avg_samples: 128.0, ips: 212.06428 samples/s, eta: 3 days, 11:10:22
[2023/10/06 17:23:08] ppocr INFO: epoch: [1/500], global_step: 140, lr: 0.000035, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.25676 s, avg_batch_cost: 0.55993 s, avg_samples: 128.0, ips: 228.59939 samples/s, eta: 3 days, 9:23:20
[2023/10/06 17:23:14] ppocr INFO: epoch: [1/500], global_step: 150, lr: 0.000037, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.35681 s, avg_batch_cost: 0.65703 s, avg_samples: 128.0, ips: 194.81524 samples/s, eta: 3 days, 8:30:56
[2023/10/06 17:23:20] ppocr INFO: epoch: [1/500], global_step: 160, lr: 0.000040, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.26879 s, avg_batch_cost: 0.56868 s, avg_samples: 128.0, ips: 225.08203 samples/s, eta: 3 days, 7:10:38
[2023/10/06 17:23:25] ppocr INFO: epoch: [1/500], global_step: 170, lr: 0.000043, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.20681 s, avg_batch_cost: 0.50916 s, avg_samples: 128.0, ips: 251.39587 samples/s, eta: 3 days, 5:37:55
[2023/10/06 17:23:31] ppocr INFO: epoch: [1/500], global_step: 180, lr: 0.000045, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.26097 s, avg_batch_cost: 0.56609 s, avg_samples: 128.0, ips: 226.11357 samples/s, eta: 3 days, 4:35:14
[2023/10/06 17:23:35] ppocr INFO: epoch: [1/500], global_step: 190, lr: 0.000048, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.15217 s, avg_batch_cost: 0.45943 s, avg_samples: 128.0, ips: 278.60882 samples/s, eta: 3 days, 3:04:06
[2023/10/06 17:23:40] ppocr INFO: epoch: [1/500], global_step: 200, lr: 0.000051, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.14376 s, avg_batch_cost: 0.44422 s, avg_samples: 128.0, ips: 288.14593 samples/s, eta: 3 days, 1:37:21
[2023/10/06 17:23:44] ppocr INFO: epoch: [1/500], global_step: 210, lr: 0.000053, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.13565 s, avg_batch_cost: 0.43672 s, avg_samples: 128.0, ips: 293.09288 samples/s, eta: 3 days, 0:16:37
[2023/10/06 17:23:49] ppocr INFO: epoch: [1/500], global_step: 220, lr: 0.000056, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.17324 s, avg_batch_cost: 0.47677 s, avg_samples: 128.0, ips: 268.47076 samples/s, eta: 2 days, 23:14:34
[2023/10/06 17:23:53] ppocr INFO: epoch: [1/500], global_step: 230, lr: 0.000059, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.07787 s, avg_batch_cost: 0.38420 s, avg_samples: 128.0, ips: 333.15772 samples/s, eta: 2 days, 21:52:49
[2023/10/06 17:23:57] ppocr INFO: epoch: [1/500], global_step: 240, lr: 0.000061, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.14407 s, avg_batch_cost: 0.45060 s, avg_samples: 128.0, ips: 284.06847 samples/s, eta: 2 days, 20:55:07
[2023/10/06 17:24:02] ppocr INFO: epoch: [1/500], global_step: 250, lr: 0.000064, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.12463 s, avg_batch_cost: 0.43017 s, avg_samples: 128.0, ips: 297.55712 samples/s, eta: 2 days, 19:56:56

Oct 06 '23 12:10 IbrarBabar009

If you change the dictionary file then I think last three layers are changed so the accuracy will start from 0.00, but if not it shouldn't one more thing while finetuning model don't use cosine use piecewise please refer this docs (I know you are training from scratch just to let you know about finetuning) https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.7/doc/doc_en/finetune_en.md

Oct 07 '23 03:10 hritikakolkar

@hritikakolkar, I use paddleOCR-2.6.1 version. But I didn't get change in Loss and accuracy

here is the epoch which is still running but not getting any change,

Need your giudence and help in this matter

[2023/10/09 10:49:12] ppocr INFO: epoch: [986/2000], global_step: 737800, lr: 0.000516, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30189 s, avg_samples: 128.0, ips: 423.98978 samples/s, eta: 2 days, 17:48:30
[2023/10/09 10:49:15] ppocr INFO: epoch: [986/2000], global_step: 737810, lr: 0.000516, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30229 s, avg_samples: 128.0, ips: 423.43968 samples/s, eta: 2 days, 17:48:27
[2023/10/09 10:49:18] ppocr INFO: epoch: [986/2000], global_step: 737820, lr: 0.000516, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.30219 s, avg_samples: 128.0, ips: 423.57238 samples/s, eta: 2 days, 17:48:24
[2023/10/09 10:49:21] ppocr INFO: epoch: [986/2000], global_step: 737830, lr: 0.000516, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.30288 s, avg_samples: 128.0, ips: 422.60871 samples/s, eta: 2 days, 17:48:21
[2023/10/09 10:49:24] ppocr INFO: epoch: [986/2000], global_step: 737840, lr: 0.000516, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.30313 s, avg_samples: 128.0, ips: 422.26233 samples/s, eta: 2 days, 17:48:17
[2023/10/09 10:49:27] ppocr INFO: epoch: [986/2000], global_step: 737850, lr: 0.000516, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.30304 s, avg_samples: 128.0, ips: 422.38229 samples/s, eta: 2 days, 17:48:14
[2023/10/09 10:49:30] ppocr INFO: epoch: [986/2000], global_step: 737860, lr: 0.000516, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.30342 s, avg_samples: 128.0, ips: 421.85269 samples/s, eta: 2 days, 17:48:11
[2023/10/09 10:49:33] ppocr INFO: epoch: [986/2000], global_step: 737870, lr: 0.000516, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30290 s, avg_samples: 128.0, ips: 422.58470 samples/s, eta: 2 days, 17:48:08
[2023/10/09 10:49:36] ppocr INFO: epoch: [986/2000], global_step: 737880, lr: 0.000516, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30299 s, avg_samples: 128.0, ips: 422.45973 samples/s, eta: 2 days, 17:48:05
[2023/10/09 10:49:39] ppocr INFO: epoch: [986/2000], global_step: 737890, lr: 0.000516, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30252 s, avg_samples: 128.0, ips: 423.10767 samples/s, eta: 2 days, 17:48:01
[2023/10/09 10:49:42] ppocr INFO: epoch: [986/2000], global_step: 737900, lr: 0.000516, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.30300 s, avg_samples: 128.0, ips: 422.44045 samples/s, eta: 2 days, 17:47:58
[2023/10/09 10:49:45] ppocr INFO: epoch: [986/2000], global_step: 737910, lr: 0.000516, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00026 s, avg_batch_cost: 0.30352 s, avg_samples: 128.0, ips: 421.72167 samples/s, eta: 2 days, 17:47:55
[2023/10/09 10:49:48] ppocr INFO: epoch: [986/2000], global_step: 737920, lr: 0.000516, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.30278 s, avg_samples: 128.0, ips: 422.74292 samples/s, eta: 2 days, 17:47:52
[2023/10/09 10:49:51] ppocr INFO: epoch: [986/2000], global_step: 737930, lr: 0.000516, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30273 s, avg_samples: 128.0, ips: 422.81723 samples/s, eta: 2 days, 17:47:49
[2023/10/09 10:49:54] ppocr INFO: epoch: [986/2000], global_step: 737940, lr: 0.000516, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.30271 s, avg_samples: 128.0, ips: 422.85233 samples/s, eta: 2 days, 17:47:45
[2023/10/09 10:49:57] ppocr INFO: epoch: [986/2000], global_step: 737950, lr: 0.000516, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00024 s, avg_batch_cost: 0.30254 s, avg_samples: 128.0, ips: 423.08513 samples/s, eta: 2 days, 17:47:42
[2023/10/09 10:50:01] ppocr INFO: epoch: [986/2000], global_step: 737960, lr: 0.000516, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30371 s, avg_samples: 128.0, ips: 421.44954 samples/s, eta: 2 days, 17:47:39
[2023/10/09 10:50:04] ppocr INFO: epoch: [986/2000], global_step: 737970, lr: 0.000516, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.30455 s, avg_samples: 128.0, ips: 420.29029 samples/s, eta: 2 days, 17:47:36
[2023/10/09 10:50:07] ppocr INFO: epoch: [986/2000], global_step: 737980, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.30442 s, avg_samples: 128.0, ips: 420.47578 samples/s, eta: 2 days, 17:47:33
[2023/10/09 10:50:10] ppocr INFO: epoch: [986/2000], global_step: 737990, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.30347 s, avg_samples: 128.0, ips: 421.79158 samples/s, eta: 2 days, 17:47:29
[2023/10/09 10:50:13] ppocr INFO: epoch: [986/2000], global_step: 738000, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30328 s, avg_samples: 128.0, ips: 422.04610 samples/s, eta: 2 days, 17:47:26
eval model:: 100%|█████████████████████████████████████████████████████████████████████████| 82/82 [00:12<00:00,  6.60it/s]
[2023/10/09 10:50:25] ppocr INFO: cur metric, acc: 0.0, norm_edit_dis: 9.58956580809911e-10, fps: 4249.327658780702
[2023/10/09 10:50:28] ppocr INFO: save best model is to ./output/v3_arabic_mobile/best_accuracy
[2023/10/09 10:50:28] ppocr INFO: best metric, acc: 0.0, is_float16: False, norm_edit_dis: 9.58956580809911e-10, fps: 4249.327658780702, best_epoch: 986
[2023/10/09 10:50:31] ppocr INFO: epoch: [986/2000], global_step: 738010, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00022 s, avg_batch_cost: 0.30209 s, avg_samples: 128.0, ips: 423.71846 samples/s, eta: 2 days, 17:47:23
[2023/10/09 10:50:34] ppocr INFO: epoch: [986/2000], global_step: 738020, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.30165 s, avg_samples: 128.0, ips: 424.32918 samples/s, eta: 2 days, 17:47:20
[2023/10/09 10:50:37] ppocr INFO: epoch: [986/2000], global_step: 738030, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.30161 s, avg_samples: 128.0, ips: 424.39154 samples/s, eta: 2 days, 17:47:16
[2023/10/09 10:50:40] ppocr INFO: epoch: [986/2000], global_step: 738040, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.30206 s, avg_samples: 128.0, ips: 423.75268 samples/s, eta: 2 days, 17:47:13
[2023/10/09 10:50:43] ppocr INFO: epoch: [986/2000], global_step: 738050, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.30286 s, avg_samples: 128.0, ips: 422.64345 samples/s, eta: 2 days, 17:47:10
[2023/10/09 10:50:47] ppocr INFO: epoch: [986/2000], global_step: 738060, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.30278 s, avg_samples: 128.0, ips: 422.75031 samples/s, eta: 2 days, 17:47:07
[2023/10/09 10:50:50] ppocr INFO: epoch: [986/2000], global_step: 738070, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.30317 s, avg_samples: 128.0, ips: 422.21096 samples/s, eta: 2 days, 17:47:04
[2023/10/09 10:50:53] ppocr INFO: epoch: [986/2000], global_step: 738080, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.30258 s, avg_samples: 128.0, ips: 423.02869 samples/s, eta: 2 days, 17:47:00
[2023/10/09 10:50:56] ppocr INFO: epoch: [986/2000], global_step: 738090, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30292 s, avg_samples: 128.0, ips: 422.55044 samples/s, eta: 2 days, 17:46:57
[2023/10/09 10:50:59] ppocr INFO: epoch: [986/2000], global_step: 738100, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.30290 s, avg_samples: 128.0, ips: 422.58493 samples/s, eta: 2 days, 17:46:54
[2023/10/09 10:51:02] ppocr INFO: epoch: [986/2000], global_step: 738110, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30281 s, avg_samples: 128.0, ips: 422.70138 samples/s, eta: 2 days, 17:46:51
[2023/10/09 10:51:05] ppocr INFO: epoch: [986/2000], global_step: 738120, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.30282 s, avg_samples: 128.0, ips: 422.68720 samples/s, eta: 2 days, 17:46:48
[2023/10/09 10:51:08] ppocr INFO: epoch: [986/2000], global_step: 738130, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.30233 s, avg_samples: 128.0, ips: 423.38455 samples/s, eta: 2 days, 17:46:44
[2023/10/09 10:51:11] ppocr INFO: epoch: [986/2000], global_step: 738140, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.30250 s, avg_samples: 128.0, ips: 423.13578 samples/s, eta: 2 days, 17:46:41
[2023/10/09 10:51:14] ppocr INFO: epoch: [986/2000], global_step: 738150, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30293 s, avg_samples: 128.0, ips: 422.53853 samples/s, eta: 2 days, 17:46:38
[2023/10/09 10:51:17] ppocr INFO: epoch: [986/2000], global_step: 738160, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30355 s, avg_samples: 128.0, ips: 421.67225 samples/s, eta: 2 days, 17:46:35
[2023/10/09 10:51:20] ppocr INFO: epoch: [986/2000], global_step: 738170, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30305 s, avg_samples: 128.0, ips: 422.37236 samples/s, eta: 2 days, 17:46:32
[2023/10/09 10:51:23] ppocr INFO: epoch: [986/2000], global_step: 738180, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00024 s, avg_batch_cost: 0.30339 s, avg_samples: 128.0, ips: 421.89443 samples/s, eta: 2 days, 17:46:28
[2023/10/09 10:51:26] ppocr INFO: epoch: [986/2000], global_step: 738190, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.30472 s, avg_samples: 128.0, ips: 420.05191 samples/s, eta: 2 days, 17:46:25
[2023/10/09 10:51:29] ppocr INFO: epoch: [986/2000], global_step: 738200, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30264 s, avg_samples: 128.0, ips: 422.94271 samples/s, eta: 2 days, 17:46:22
[2023/10/09 10:51:32] ppocr INFO: epoch: [986/2000], global_step: 738210, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30328 s, avg_samples: 128.0, ips: 422.05320 samples/s, eta: 2 days, 17:46:19
[2023/10/09 10:51:35] ppocr INFO: epoch: [986/2000], global_step: 738220, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30302 s, avg_samples: 128.0, ips: 422.41187 samples/s, eta: 2 days, 17:46:16
[2023/10/09 10:51:38] ppocr INFO: epoch: [986/2000], global_step: 738230, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30276 s, avg_samples: 128.0, ips: 422.77601 samples/s, eta: 2 days, 17:46:12
[2023/10/09 10:51:41] ppocr INFO: epoch: [986/2000], global_step: 738240, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30271 s, avg_samples: 128.0, ips: 422.84457 samples/s, eta: 2 days, 17:46:09
[2023/10/09 10:51:44] ppocr INFO: epoch: [986/2000], global_step: 738250, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30276 s, avg_samples: 128.0, ips: 422.77341 samples/s, eta: 2 days, 17:46:06
[2023/10/09 10:51:47] ppocr INFO: epoch: [986/2000], global_step: 738260, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.30287 s, avg_samples: 128.0, ips: 422.62172 samples/s, eta: 2 days, 17:46:03
[2023/10/09 10:51:50] ppocr INFO: epoch: [986/2000], global_step: 738270, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.30265 s, avg_samples: 128.0, ips: 422.93531 samples/s, eta: 2 days, 17:46:00
[2023/10/09 10:51:53] ppocr INFO: epoch: [986/2000], global_step: 738280, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.30287 s, avg_samples: 128.0, ips: 422.62149 samples/s, eta: 2 days, 17:45:56
[2023/10/09 10:51:56] ppocr INFO: epoch: [986/2000], global_step: 738290, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.30326 s, avg_samples: 128.0, ips: 422.07692 samples/s, eta: 2 days, 17:45:53
[2023/10/09 10:51:59] ppocr INFO: epoch: [986/2000], global_step: 738300, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.30239 s, avg_samples: 128.0, ips: 423.29682 samples/s, eta: 2 days, 17:45:50
[2023/10/09 10:52:02] ppocr INFO: epoch: [986/2000], global_step: 738310, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.30232 s, avg_samples: 128.0, ips: 423.38682 samples/s, eta: 2 days, 17:45:47
[2023/10/09 10:52:05] ppocr INFO: epoch: [986/2000], global_step: 738320, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.30306 s, avg_samples: 128.0, ips: 422.35445 samples/s, eta: 2 days, 17:45:43
[2023/10/09 10:52:08] ppocr INFO: epoch: [986/2000], global_step: 738330, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30356 s, avg_samples: 128.0, ips: 421.66483 samples/s, eta: 2 days, 17:45:40
[2023/10/09 10:52:11] ppocr INFO: epoch: [986/2000], global_step: 738340, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.30271 s, avg_samples: 128.0, ips: 422.84241 samples/s, eta: 2 days, 17:45:37
[2023/10/09 10:52:14] ppocr INFO: epoch: [986/2000], global_step: 738350, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30502 s, avg_samples: 128.0, ips: 419.64829 samples/s, eta: 2 days, 17:45:34
[2023/10/09 10:52:17] ppocr INFO: epoch: [986/2000], global_step: 738360, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.30270 s, avg_samples: 128.0, ips: 422.86582 samples/s, eta: 2 days, 17:45:31
[2023/10/09 10:52:21] ppocr INFO: epoch: [986/2000], global_step: 738370, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.30477 s, avg_samples: 128.0, ips: 419.98366 samples/s, eta: 2 days, 17:45:28
[2023/10/09 10:52:24] ppocr INFO: epoch: [986/2000], global_step: 738380, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.30622 s, avg_samples: 128.0, ips: 417.99735 samples/s, eta: 2 days, 17:45:24
[2023/10/09 10:52:27] ppocr INFO: epoch: [986/2000], global_step: 738390, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30559 s, avg_samples: 128.0, ips: 418.86536 samples/s, eta: 2 days, 17:45:21
[2023/10/09 10:52:30] ppocr INFO: epoch: [986/2000], global_step: 738400, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30374 s, avg_samples: 128.0, ips: 421.41771 samples/s, eta: 2 days, 17:45:18
[2023/10/09 10:52:33] ppocr INFO: epoch: [986/2000], global_step: 738410, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.30292 s, avg_samples: 128.0, ips: 422.55925 samples/s, eta: 2 days, 17:45:15
[2023/10/09 10:52:36] ppocr INFO: epoch: [986/2000], global_step: 738420, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30422 s, avg_samples: 128.0, ips: 420.74566 samples/s, eta: 2 days, 17:45:12
[2023/10/09 10:52:39] ppocr INFO: epoch: [986/2000], global_step: 738430, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30416 s, avg_samples: 128.0, ips: 420.82946 samples/s, eta: 2 days, 17:45:08
[2023/10/09 10:52:42] ppocr INFO: epoch: [986/2000], global_step: 738440, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.30307 s, avg_samples: 128.0, ips: 422.33913 samples/s, eta: 2 days, 17:45:05
[2023/10/09 10:52:45] ppocr INFO: epoch: [986/2000], global_step: 738450, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.30496 s, avg_samples: 128.0, ips: 419.72490 samples/s, eta: 2 days, 17:45:02
[2023/10/09 10:52:48] ppocr INFO: epoch: [986/2000], global_step: 738460, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.30318 s, avg_samples: 128.0, ips: 422.19834 samples/s, eta: 2 days, 17:44:59
[2023/10/09 10:52:51] ppocr INFO: epoch: [986/2000], global_step: 738470, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.30306 s, avg_samples: 128.0, ips: 422.35690 samples/s, eta: 2 days, 17:44:56
[2023/10/09 10:52:54] ppocr INFO: epoch: [986/2000], global_step: 738480, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.30258 s, avg_samples: 128.0, ips: 423.02652 samples/s, eta: 2 days, 17:44:52
[2023/10/09 10:52:57] ppocr INFO: epoch: [986/2000], global_step: 738490, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.30265 s, avg_samples: 128.0, ips: 422.93601 samples/s, eta: 2 days, 17:44:49
[2023/10/09 10:53:00] ppocr INFO: epoch: [986/2000], global_step: 738500, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30329 s, avg_samples: 128.0, ips: 422.03727 samples/s, eta: 2 days, 17:44:46
[2023/10/09 10:53:03] ppocr INFO: epoch: [986/2000], global_step: 738510, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00025 s, avg_batch_cost: 0.30261 s, avg_samples: 128.0, ips: 422.98566 samples/s, eta: 2 days, 17:44:43
[2023/10/09 10:53:04] ppocr INFO: epoch: [986/2000], global_step: 738514, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00003 s, avg_batch_cost: 0.12026 s, avg_samples: 51.2, ips: 425.75350 samples/s, eta: 2 days, 17:44:41
[2023/10/09 10:53:16] ppocr INFO: save model in ./output/v3_arabic_mobile/latest
[2023/10/09 10:53:18] ppocr INFO: epoch: [987/2000], global_step: 738520, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 1.18660 s, avg_batch_cost: 1.37187 s, avg_samples: 76.8, ips: 55.98194 samples/s, eta: 2 days, 17:44:52
[2023/10/09 10:53:21] ppocr INFO: epoch: [987/2000], global_step: 738530, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.30288 s, avg_samples: 128.0, ips: 422.60778 samples/s, eta: 2 days, 17:44:49
[2023/10/09 10:53:24] ppocr INFO: epoch: [987/2000], global_step: 738540, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.30316 s, avg_samples: 128.0, ips: 422.22570 samples/s, eta: 2 days, 17:44:45
[2023/10/09 10:53:27] ppocr INFO: epoch: [987/2000], global_step: 738550, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30235 s, avg_samples: 128.0, ips: 423.35547 samples/s, eta: 2 days, 17:44:42
[2023/10/09 10:53:30] ppocr INFO: epoch: [987/2000], global_step: 738560, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30317 s, avg_samples: 128.0, ips: 422.20528 samples/s, eta: 2 days, 17:44:39
[2023/10/09 10:53:33] ppocr INFO: epoch: [987/2000], global_step: 738570, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30248 s, avg_samples: 128.0, ips: 423.17223 samples/s, eta: 2 days, 17:44:36
[2023/10/09 10:53:36] ppocr INFO: epoch: [987/2000], global_step: 738580, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30317 s, avg_samples: 128.0, ips: 422.21212 samples/s, eta: 2 days, 17:44:33
[2023/10/09 10:53:39] ppocr INFO: epoch: [987/2000], global_step: 738590, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.30247 s, avg_samples: 128.0, ips: 423.17727 samples/s, eta: 2 days, 17:44:29
[2023/10/09 10:53:42] ppocr INFO: epoch: [987/2000], global_step: 738600, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.30270 s, avg_samples: 128.0, ips: 422.85809 samples/s, eta: 2 days, 17:44:26
[2023/10/09 10:53:45] ppocr INFO: epoch: [987/2000], global_step: 738610, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30241 s, avg_samples: 128.0, ips: 423.26889 samples/s, eta: 2 days, 17:44:23
[2023/10/09 10:53:48] ppocr INFO: epoch: [987/2000], global_step: 738620, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.30243 s, avg_samples: 128.0, ips: 423.24219 samples/s, eta: 2 days, 17:44:20
[2023/10/09 10:53:51] ppocr INFO: epoch: [987/2000], global_step: 738630, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30263 s, avg_samples: 128.0, ips: 422.96077 samples/s, eta: 2 days, 17:44:16
[2023/10/09 10:53:54] ppocr INFO: epoch: [987/2000], global_step: 738640, lr: 0.000515, acc: 0.000000, norm_edit_dis: 0.000000, CTCLoss: nanxxx, SARLoss: nanxxx, loss: nanxxx, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.30238 s, avg_samples: 128.0, ips: 423.31020 samples/s, eta: 2 days, 17:44:13

What should I have to do to solve the problem? is it possible for you to give me command and versions details and dependencies that you used for training?

Oct 09 '23 05:10 IbrarBabar009

I don't know what is the problem. But you have to dig down se whether the image is being read, is label being read (might be the problem) other than that I can't help.

Oct 09 '23 11:10 hritikakolkar

Sad to here this from you.

Oct 09 '23 12:10 IbrarBabar009

@hritikakolkar I have an update, I tried 3 versions of PaddleOCR (2.7 2.6, 2.5) but the issue is same, So, I just started training PaddleOCR on CPU, and it began to show a decrease in loss.

Could this possibly be due to the CUDA version? Do you have any insights on this?

Oct 10 '23 05:10 IbrarBabar009

Okay, I don't know whether that be the problem here is what version I use.

paddle-bfloat==0.1.7
paddleocr==2.6.1.3
paddlepaddle-gpu==2.4.2.post117

if you want to install use this command

python -m pip install paddlepaddle-gpu==2.4.2.post117 -f https://www.paddlepaddle.org.cn/whl/linux/mkl/avx/stable.html

Oct 10 '23 07:10 hritikakolkar

Thank you so much, I found it it is because of CUDA version I currently have CUDA 12.1 and PaddleOCR support CUDA 11.7 or lower then that. I am facing this issue because of that reason.

Oct 10 '23 13:10 IbrarBabar009

oh good

Oct 11 '23 04:10 hritikakolkar

@hritikakolkar Bro, can you share your experiences? Have you trained PaddleOCR on an Arabic dataset, and did you finetune it, or did you train it from scratch?

I'm asking these questions because I trained PaddleOCR on an Arabic dataset and achieved an accuracy of 83.93%, which I find somewhat unsatisfactory.

I used the following command to train PaddleOCR with finetuning:

python3 tools/train.py -c configs/rec/PP-OCRv3/multi_language/arabic_PP-OCRv3_rec.yml -o Global.pretrained_model=/additional_drive/ibrar/PaddleOCR/pretrain_models/arabic/arabic_PP-OCRv3_rec_train/best_accuracy.pdparams

I've already shared my YML file with you. I'm a bit confused because when I test images with the pre-trained model, it performs very well. However, when I test it on my trained model, the performance isn't as good.

Do you have any insights into why this might be happening? Is this the right way to start finetuning the model for training?"

Oct 13 '23 06:10 IbrarBabar009

If you are not getting desired accuracy then there might be two problems, the dataset distribution is different from real world data you are inferencing on, otherwise training hyperparameters like batch size, learning rate are important. Please also try to infer on multiple epochs for example check how model worked on epoch_10, epoch_11, for doing this you have to save model at every epoch while training.

Oct 13 '23 07:10 hritikakolkar

The real problem is of data, paddleocr guys haven't shared the dataset in which they trained arabic ocr model so that's the issue

Oct 13 '23 07:10 hritikakolkar

Hmm, that makes sense. Thank you so much. I will let you know after conducting these experiments.

Oct 13 '23 10:10 IbrarBabar009

Thank you so much, I found it it is because of CUDA version I currently have CUDA 12.1 and PaddleOCR support CUDA 11.7 or lower then that. I am facing this issue because of that reason.

I'm also cuda12.1, and i installed paddle2.5.1post12.0, it works well.you can have a try

Oct 26 '23 13:10 data-ant

@data-ant Thank you so much. You are right. I was initially trying to install a lower version that was not compatible with CUDA 12.1. However, I have since downgraded my CUDA to version 11.7, and it works fine now.

Subsequently, I realized that the issue was not related to the CUDA version but rather to the PaddleOCR version.

Oct 26 '23 13:10 IbrarBabar009

我已收到，谢谢。祝您诸事顺利!

Dec 01 '23 09:12 data-ant

@hritikakolkar @WenmuZhou @dyning @LDOUBLEV @tink2123 @MissPenguin @Topdu @Evezerest @littletomatodonkey @andyjpaddle @weisy11 @D-DanielYang @Topdu @sdcb @ZeyuChen @data-ant @bingooo please help?????

I am using PaddleOCR for Arabic dataset recognition, and I have achieved an accuracy of 97.76%. However, I am encountering issues with error rates, specifically on Arabic dates like the example provided below:

When I input such images into my trained PaddleOCR recognition model, it occasionally misses 2, 3, or 4 words or adds extra integers. Despite attempting to train PaddleOCR separately on dates with a labeled dataset of approximately 60,000 images, each with exact and correct ground truth, the accuracy does not seem to improve.

I am seeking assistance to understand why PaddleOCR is struggling with Arabic numbers and if there is any way to enhance its performance. Additionally, I am open to exploring alternative methods to address this issue. Can anyone provide insights or suggestions on improving the accuracy of PaddleOCR for recognizing Arabic numbers in date formats?"

Dec 13 '23 07:12 IbrarBabar009

@IbrarBabar009 - would you be able to share the finetuned Arabic model?

Also have you done any fine-tuning on Arabic detection, as I am using paddle on scene-text detection & recognition in Arabic?

Dec 19 '23 09:12 connorourke

It will work fine just add general data along with dates also you needed to add Arabic 0 inn the dictionary

Dec 20 '23 22:12 hritikakolkar

PaddleOCR PaddleOCR copied to clipboard

consistently getting Loss = nanxxx and accuracy 0.000000 even for running on 1500 epoches on arabic dataset.

PaddleOCR
PaddleOCR copied to clipboard