kraken
kraken copied to clipboard
Published
20 hours ago •
mittagessen
mittagessen
kraken 5.2.9 segmentation training no lines
I try to train a segmentation model with page-xml data, at start the segmenter shows me the regions and line types but when using the model no lines is detected at all!
(kraken-5.2.9) incognito@DESKTOP-NHKR7QL:~/kraken-train/102_Petrov_isbach$ ketos segtrain -d cuda:0 -f page -t output.txt -q early -cl --min-epochs 40 -o /home/incognito/kraken-train/102_Petrov_isbach/seg_v2/isbach_seg_v2
Training line types:
textline 2 1034
Training region types:
textzone 3 19
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
You are using a CUDA device ('NVIDIA GeForce RTX 4070') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
┏━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ ┃ Name ┃ Type ┃ Params ┃ In sizes ┃ Out sizes ┃
┡━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 0 │ net │ MultiParamSequential │ 1.3 M │ [1, 3, 1800, 300] │ [[1, 4, 450, 75], '?'] │
│ 1 │ net.C_0 │ ActConv2D │ 9.5 K │ [[1, 3, 1800, 300], '?'] │ [[1, 64, 900, 150], '?'] │
│ 2 │ net.Gn_1 │ GroupNorm │ 128 │ [[1, 64, 900, 150], '?', '?'] │ [[1, 64, 900, 150], '?'] │
│ 3 │ net.C_2 │ ActConv2D │ 73.9 K │ [[1, 64, 900, 150], '?', '?'] │ [[1, 128, 450, 75], '?'] │
│ 4 │ net.Gn_3 │ GroupNorm │ 256 │ [[1, 128, 450, 75], '?', '?'] │ [[1, 128, 450, 75], '?'] │
│ 5 │ net.C_4 │ ActConv2D │ 147 K │ [[1, 128, 450, 75], '?', '?'] │ [[1, 128, 450, 75], '?'] │
│ 6 │ net.Gn_5 │ GroupNorm │ 256 │ [[1, 128, 450, 75], '?', '?'] │ [[1, 128, 450, 75], '?'] │
│ 7 │ net.C_6 │ ActConv2D │ 295 K │ [[1, 128, 450, 75], '?', '?'] │ [[1, 256, 450, 75], '?'] │
│ 8 │ net.Gn_7 │ GroupNorm │ 512 │ [[1, 256, 450, 75], '?', '?'] │ [[1, 256, 450, 75], '?'] │
│ 9 │ net.C_8 │ ActConv2D │ 590 K │ [[1, 256, 450, 75], '?', '?'] │ [[1, 256, 450, 75], '?'] │
│ 10 │ net.Gn_9 │ GroupNorm │ 512 │ [[1, 256, 450, 75], '?', '?'] │ [[1, 256, 450, 75], '?'] │
│ 11 │ net.L_10 │ TransposedSummarizingRNN │ 74.2 K │ [[1, 256, 450, 75], '?', '?'] │ [[1, 64, 450, 75], '?'] │
│ 12 │ net.L_11 │ TransposedSummarizingRNN │ 25.1 K │ [[1, 64, 450, 75], '?', '?'] │ [[1, 64, 450, 75], '?'] │
│ 13 │ net.C_12 │ ActConv2D │ 2.1 K │ [[1, 64, 450, 75], '?', '?'] │ [[1, 32, 450, 75], '?'] │
│ 14 │ net.Gn_13 │ GroupNorm │ 64 │ [[1, 32, 450, 75], '?', '?'] │ [[1, 32, 450, 75], '?'] │
│ 15 │ net.L_14 │ TransposedSummarizingRNN │ 16.9 K │ [[1, 32, 450, 75], '?', '?'] │ [[1, 64, 450, 75], '?'] │
│ 16 │ net.L_15 │ TransposedSummarizingRNN │ 25.1 K │ [[1, 64, 450, 75], '?', '?'] │ [[1, 64, 450, 75], '?'] │
│ 17 │ net.l_16 │ ActConv2D │ 260 │ [[1, 64, 450, 75], '?', '?'] │ [[1, 4, 450, 75], '?'] │
│ 18 │ val_px_accuracy │ MultilabelAccuracy │ 0 │ ? │ ? │
│ 19 │ val_mean_accuracy │ MultilabelAccuracy │ 0 │ ? │ ? │
│ 20 │ val_mean_iu │ MultilabelJaccardIndex │ 0 │ ? │ ? │
│ 21 │ val_freq_iu │ MultilabelJaccardIndex │ 0 │ ? │ ? │
└────┴───────────────────┴──────────────────────────┴────────┴───────────────────────────────┴──────────────────────────┘
Trainable params: 1.3 M
Non-trainable params: 0
Total params: 1.3 M
Total estimated model params size (MB): 5
stage 0/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:09 • 0:00:00 2.00it/s val_accuracy: 0.878 val_mean_acc: 0.878 val_mean_iu: 0.124 val_freq_iu: 0.415 early_stopping: 0/10 0.12395
stage 1/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.93it/s val_accuracy: 0.927 val_mean_acc: 0.927 val_mean_iu: 0.144 val_freq_iu: 0.482 early_stopping: 0/10 0.14409
stage 2/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.91it/s val_accuracy: 0.945 val_mean_acc: 0.945 val_mean_iu: 0.161 val_freq_iu: 0.539 early_stopping: 0/10 0.16118
stage 3/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.95it/s val_accuracy: 0.975 val_mean_acc: 0.975 val_mean_iu: 0.218 val_freq_iu: 0.728 early_stopping: 0/10 0.21762
stage 4/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.94it/s val_accuracy: 0.976 val_mean_acc: 0.976 val_mean_iu: 0.221 val_freq_iu: 0.739 early_stopping: 0/10 0.22080
stage 5/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:09 • 0:00:00 1.90it/s val_accuracy: 0.975 val_mean_acc: 0.975 val_mean_iu: 0.217 val_freq_iu: 0.725 early_stopping: 1/10 0.22080
stage 6/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.93it/s val_accuracy: 0.976 val_mean_acc: 0.976 val_mean_iu: 0.222 val_freq_iu: 0.743 early_stopping: 0/10 0.22191
stage 7/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.94it/s val_accuracy: 0.976 val_mean_acc: 0.976 val_mean_iu: 0.220 val_freq_iu: 0.737 early_stopping: 1/10 0.22191
stage 8/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.94it/s val_accuracy: 0.977 val_mean_acc: 0.977 val_mean_iu: 0.222 val_freq_iu: 0.744 early_stopping: 0/10 0.22241
stage 9/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:09 • 0:00:00 1.90it/s val_accuracy: 0.976 val_mean_acc: 0.976 val_mean_iu: 0.221 val_freq_iu: 0.740 early_stopping: 1/10 0.22241
stage 10/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:09 • 0:00:00 1.93it/s val_accuracy: 0.976 val_mean_acc: 0.976 val_mean_iu: 0.220 val_freq_iu: 0.735 early_stopping: 2/10 0.22241
stage 11/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.93it/s val_accuracy: 0.977 val_mean_acc: 0.977 val_mean_iu: 0.225 val_freq_iu: 0.752 early_stopping: 0/10 0.22481
stage 12/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.92it/s val_accuracy: 0.977 val_mean_acc: 0.977 val_mean_iu: 0.225 val_freq_iu: 0.752 early_stopping: 1/10 0.22481
stage 13/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.88it/s val_accuracy: 0.977 val_mean_acc: 0.977 val_mean_iu: 0.225 val_freq_iu: 0.754 early_stopping: 0/10 0.22528
stage 14/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.95it/s val_accuracy: 0.977 val_mean_acc: 0.977 val_mean_iu: 0.225 val_freq_iu: 0.753 early_stopping: 1/10 0.22528
stage 15/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.89it/s val_accuracy: 0.976 val_mean_acc: 0.976 val_mean_iu: 0.222 val_freq_iu: 0.742 early_stopping: 2/10 0.22528
stage 16/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:09 • 0:00:00 1.91it/s val_accuracy: 0.977 val_mean_acc: 0.977 val_mean_iu: 0.225 val_freq_iu: 0.752 early_stopping: 3/10 0.22528
stage 17/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.90it/s val_accuracy: 0.977 val_mean_acc: 0.977 val_mean_iu: 0.225 val_freq_iu: 0.753 early_stopping: 4/10 0.22528
stage 18/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.94it/s val_accuracy: 0.977 val_mean_acc: 0.977 val_mean_iu: 0.225 val_freq_iu: 0.753 early_stopping: 5/10 0.22528
stage 19/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.93it/s val_accuracy: 0.976 val_mean_acc: 0.976 val_mean_iu: 0.220 val_freq_iu: 0.737 early_stopping: 6/10 0.22528
stage 20/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.93it/s val_accuracy: 0.974 val_mean_acc: 0.974 val_mean_iu: 0.214 val_freq_iu: 0.716 early_stopping: 7/10 0.22528
stage 21/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.95it/s val_accuracy: 0.975 val_mean_acc: 0.975 val_mean_iu: 0.218 val_freq_iu: 0.731 early_stopping: 8/10 0.22528
stage 22/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.94it/s val_accuracy: 0.978 val_mean_acc: 0.978 val_mean_iu: 0.227 val_freq_iu: 0.760 early_stopping: 0/10 0.22707
stage 23/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.95it/s val_accuracy: 0.977 val_mean_acc: 0.977 val_mean_iu: 0.225 val_freq_iu: 0.752 early_stopping: 1/10 0.22707
stage 24/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.94it/s val_accuracy: 0.978 val_mean_acc: 0.978 val_mean_iu: 0.227 val_freq_iu: 0.761 early_stopping: 0/10 0.22729
stage 25/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.93it/s val_accuracy: 0.979 val_mean_acc: 0.979 val_mean_iu: 0.228 val_freq_iu: 0.765 early_stopping: 0/10 0.22842
stage 26/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:09 • 0:00:00 1.90it/s val_accuracy: 0.978 val_mean_acc: 0.978 val_mean_iu: 0.226 val_freq_iu: 0.757 early_stopping: 1/10 0.22842
stage 27/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.92it/s val_accuracy: 0.977 val_mean_acc: 0.977 val_mean_iu: 0.225 val_freq_iu: 0.753 early_stopping: 2/10 0.22842
stage 28/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.94it/s val_accuracy: 0.975 val_mean_acc: 0.975 val_mean_iu: 0.220 val_freq_iu: 0.735 early_stopping: 3/10 0.22842
stage 29/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.93it/s val_accuracy: 0.977 val_mean_acc: 0.977 val_mean_iu: 0.225 val_freq_iu: 0.755 early_stopping: 4/10 0.22842
stage 30/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.94it/s val_accuracy: 0.977 val_mean_acc: 0.977 val_mean_iu: 0.225 val_freq_iu: 0.752 early_stopping: 5/10 0.22842
stage 31/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.93it/s val_accuracy: 0.976 val_mean_acc: 0.976 val_mean_iu: 0.222 val_freq_iu: 0.744 early_stopping: 6/10 0.22842
stage 32/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.93it/s val_accuracy: 0.975 val_mean_acc: 0.975 val_mean_iu: 0.220 val_freq_iu: 0.736 early_stopping: 7/10 0.22842
stage 33/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.93it/s val_accuracy: 0.976 val_mean_acc: 0.976 val_mean_iu: 0.221 val_freq_iu: 0.738 early_stopping: 8/10 0.22842
stage 34/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.93it/s val_accuracy: 0.977 val_mean_acc: 0.977 val_mean_iu: 0.224 val_freq_iu: 0.750 early_stopping: 9/10 0.22842
stage 35/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.94it/s val_accuracy: 0.950 val_mean_acc: 0.950 val_mean_iu: 0.169 val_freq_iu: 0.565 early_stopping: 10/10 0.22842
stage 36/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/17 0:00:00 • -:--:-- 0.00it/s early_stopping: 10/10 0.22842Trainer was signaled to stop but the required `min_epochs=40` or `min_steps=None` has not been met. Training will continue...
stage 36/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.93it/s val_accuracy: 0.977 val_mean_acc: 0.977 val_mean_iu: 0.223 val_freq_iu: 0.745 early_stopping: 11/10 0.22842
stage 37/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.92it/s val_accuracy: 0.977 val_mean_acc: 0.977 val_mean_iu: 0.225 val_freq_iu: 0.753 early_stopping: 12/10 0.22842
stage 38/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:09 • 0:00:00 1.89it/s val_accuracy: 0.977 val_mean_acc: 0.977 val_mean_iu: 0.223 val_freq_iu: 0.747 early_stopping: 13/10 0.22842
stage 39/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17/17 0:00:08 • 0:00:00 1.95it/s val_accuracy: 0.976 val_mean_acc: 0.976 val_mean_iu: 0.223 val_freq_iu: 0.746 early_stopping: 14/10 0.22842
Moving best model /home/incognito/kraken-train/102_Petrov_isbach/seg_v2/isbach_seg_v2_25.mlmodel (0.22842147946357727) to /home/incognito/kraken-train/102_Petrov_isbach/seg_v2/isbach_seg_v2_best.mlmodel
Result:
