RFC: Improve positioning of symbol bounding boxes

[p12tic]

This PR improves the positions of symbol bounding boxes in cases when LSTM model is used. Up to 20 times less errors have been observed in sample images.

This PR still requires potentially significant amount of work, please let me know whether the approach is sensible in principle and if the PR makes sense I will spend time polishing it.

---

When using LSTM models the accuracy of character bounding boxes is low with many blobs assigned to wrong characters. This is caused by the fact that LSTM model output produces only approximate character positions without boundary data. As a result the input blobs cannot be accurately mapped to characters and which compromises the accuracy of character bounding boxes.

Currently this problem is solved as follows. The character boundaries are computed according to the character positions from the LSTM output by placing the boundaries at the middle between two character positions. The blobs are then assigned according to which character the center of the blob falls to. In other words the blobs are assigned to the nearest characters.

This unfortunately produces a lot of errors because the character positions in the LSTM output have a tendency to drift, thus the nearest character is often not the right one.

Fortunately while the LSTM model produces approximate positions, the blob boundaries produced by the regular segmenter are pretty good. Most of the time a single blob corresponds to a single character and vice-versa.

The above is used to create an optimization algorithm that treats the output of the regular segmenter as a template to which LSTM model output is matched. The selection of best match is done by assigning each unwanted property of the outcome a cost and then minimizing the total cost of the solution.

This reliably solves the most frequent error present in the current solution when blobs are simply assigned to wrong character. As a result the current algorithm produces up to 20 times less errors.

This can be further improved because the root cause of most of the remaining errors is the segmenter producing single blobs for multiple characters. The algorithm could be improved by biasing it to split the blobs in the places where the segmenter makes errors often, such as near character "t".

Fixes https://github.com/tesseract-ocr/tesseract/issues/1712.

---

One of the example images I've used:

Before this PR, tesseract produced 116 errors in determining character bounding boxes (could be inaccurate due to manual counting)

After this PR, tesseract produced only 5 errors in determining character bounding boxes.

[p12tic]

@stweil Just a friendly ping :-)

[rmast]

I've done some testing with this branch merged to the current main. It's still not perfect. With oem 0: With LSTM:

Original image:

Program code of bounding boxes:

import tesserocr
import csv
import numpy as np
import cv2
import pytesseract
from PIL import Image
import matplotlib as mpl

import matplotlib.pyplot as plt
img3 = cv2.imread('/home/rmast/plaatjes/out1496-3078-212-39.png')
h, w, _ = img3.shape # assumes color image
pytess_result = pytesseract.image_to_boxes(img3, lang='nld+lat+Latin+eng',
        config="--psm 7 -c tessedit_create_boxfile=1", output_type=pytesseract.Output.DICT)
        #config="--psm 7 --oem 0 -c tessedit_create_boxfile=1", output_type=pytesseract.Output.DICT)
print(pytess_result)
for j in range(0, len(pytess_result["char"])):
    left = pytess_result["left"][j]
    bottom = pytess_result["bottom"][j]
    right = pytess_result["right"][j]
    top = pytess_result["top"][j]
    cv2.rectangle(img3, (left, h - top - 1), (right, h - bottom - 1), (255, 0, 0), 1)
mpl.use('tkAgg')
plt.imshow(img3)
plt.show()

oem 0:

With LSTM:

Original image:

Without your patch:

[lv-saharan]

why still not merged

[rmast]

I guess lack of testing capacity for core functionality. Since I discovered that and some other low level segmentation bugs my focus has shifted to EasyOCR.

Verzonden vanaf Outlook voor Androidhttps://aka.ms/AAb9ysg

---

From: 海刚 @.> Sent: Thursday, October 26, 2023 7:52:16 AM To: tesseract-ocr/tesseract @.> Cc: rmast @.>; Comment @.> Subject: Re: [tesseract-ocr/tesseract] RFC: Improve positioning of symbol bounding boxes (PR #3787)

why still not merged

― Reply to this email directly, view it on GitHubhttps://github.com/tesseract-ocr/tesseract/pull/3787#issuecomment-1780457429, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAZPZ5RPVKU5GD3FJM2I7HDYBH3BBAVCNFSM5TBLNP7KU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCNZYGA2DKNZUGI4Q. You are receiving this because you commented.Message ID: @.***>