Error with OpenCV while creating the image_generators

[plrevolve]

I'm trying to use your google colab notebook to train a new model with a different alphabet. I use:

• Tensorflow v.2.3.0
• opencv-python v.4.3.0.36

The problem : I'm stuck at the step where it creates the image_generators structure. More accurately, just after setting up the image_generators at this line:

# See what the first validation image looks like.
image, lines = next(image_generators[1])

Here is the error that I get:

error                                     Traceback (most recent call last)
<ipython-input-19-1c5edc822479> in <module>
     34 
     35 # See what the first validation image looks like.
---> 36 image, lines = next(image_generators[1])
     37 text = keras_ocr.data_generation.convert_lines_to_paragraph(lines)
     38 print("The first generated validation image (below) contains:", text)

~/.local/lib/python3.8/site-packages/keras_ocr/data_generation.py in get_image_generator(height, width, font_groups, 
text_generator, font_size, backgrounds, background_crop_mode, rotationX, rotationY, rotationZ, margin, use_ligatures, augmenter, draw_contour, draw_contour_text)
    555                                            width=width,
    556                                            height=height,
--> 557                                            mode=background_crop_mode)
    558         permitted_contour, isDark = get_maximum_uniform_contour(image=current_background,
    559                                                                 fontsize=current_font_size,

~/.local/lib/python3.8/site-packages/keras_ocr/data_generation.py in get_maximum_uniform_contour(image, fontsize, margin)
     66     """
     67     if margin > 0:
---> 68         image = image[margin:-margin, margin:-margin]
     69     gray = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)
     70     blurred = cv2.blur(src=gray, ksize=(fontsize // 2, fontsize // 2))

error: OpenCV(4.3.0) /io/opencv/modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'

I haven't change any line of your code. Could you tell me which version of tensorflow and OpenCV you used while making Keras-ocr ?

[faustomorales]

I'm not sure what's going on here -- but you could start debugging by taking a peek at what's going on inside get_maximum_uniform_contour by doing something like the following. Note the lines between BEGIN DEBUGGING and END DEBUGGING.

def debug_get_maximum_uniform_contour(image, fontsize, margin=0):
    """Get the largest possible contour of light or
    dark area in an image.
    Args:
        image: The image in which to find a contiguous area.
        fontsize: The fontsize for text. Will be used for blurring
            and for determining useful areas.
        margin: The minimum margin required around the image.
    Returns:
        A (contour, isDark) tuple. If no contour is found, both
        entries will be None.
    """
    if margin > 0:
        image = image[margin:-margin, margin:-margin]
    ## BEGIN DEBUGGING
    print('image shape', image.shape)
    ## END DEBUGGING
    gray = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)
    blurred = cv2.blur(src=gray, ksize=(fontsize // 2, fontsize // 2))
    _, threshold = cv2.threshold(src=blurred, thresh=255 / 2, maxval=255, type=cv2.THRESH_BINARY)
    contoursDark = cv2.findContours(255 - threshold,
                                    mode=cv2.RETR_TREE,
                                    method=cv2.CHAIN_APPROX_SIMPLE)[-2]
    contoursLight = cv2.findContours(threshold, mode=cv2.RETR_TREE,
                                     method=cv2.CHAIN_APPROX_SIMPLE)[-2]
    areasDark = list(map(cv2.contourArea, contoursDark))
    areasLight = list(map(cv2.contourArea, contoursLight))
    maxDarkArea = max(areasDark) if areasDark else 0
    maxLightArea = max(areasLight) if areasLight else 0

    if max(maxDarkArea, maxLightArea) < (4 * fontsize)**2:
        return None, None

    contour = None
    isDark = None
    if areasDark and (not areasLight or maxDarkArea >= maxLightArea):
        contour = contoursDark[np.argmax(areasDark)]
        isDark = True
    else:
        contour = contoursLight[np.argmax(areasLight)]
        isDark = False
    if contour is not None:
        contour += margin
    return contour, isDark

# Make keras-ocr use your debugging function
keras_ocr.data_generation.get_maximum_uniform_contour = debug_get_maximum_uniform_contour

# Trigger error here and investigate what `image` is just before the exception.

I don't think this is an OpenCV issue because the OpenCV functions used here are rudimentary. I assume you're using the example dataset and not your own?

[plrevolve]

Thanks for your answer,

I already tried to debug like this but it looks like the file don't change as it didn't print anything in the console, even if I re-import the module.

I'm using indeed the example dataset and image generator. I've just changed some parameters of the image generator :

image_generators = [
    keras_ocr.data_generation.get_image_generator(
        height=100,
        width=2600,
        text_generator=text_generator,
        font_groups={
            alphabet: current_fonts
        },
        backgrounds=current_backgrounds,
        font_size=(60, 120),
        margin=50,
        rotationX=(0, 0),
        rotationY=(0, 0),
        rotationZ=(0, 0)
    )  for current_fonts, current_backgrounds in zip(
        font_splits,
        background_splits
    )
] 

Also, I changed the alphabet for special characters I need.

alphabet = string.digits + string.ascii_letters + '#-°:'

[kadarshj]

I'm trying to use your google Colab Notebook to train a new model with a different alphabet i.e. included some punctuation. Version I'm using below for tensorflow & opencv:

• Tensorflow v.2.5.0
• opencv-python v.4.5.2

The problem : While trying your notebook I got stuck at the step where it creates the image_generators structure. More accurately, just after setting up the image_generators at this line:

# See what the first validation image looks like. image, lines = next(image_generators[1])

this is the error i'm getting below:

error Traceback (most recent call last) <ipython-input-14-4e7ecca7baec> in <module>() 31 32 # See what the first validation image looks like. ---> 33 image, lines = next(image_generators[1]) 34 text = keras_ocr.data_generation.convert_lines_to_paragraph(lines) 35 print('The first generated validation image (below) contains:', text) `/usr/local/lib/python3.7/dist-packages/keras_ocr/data_generation.py in get_image_generator(height, width, font_groups, text_generator, font_size, backgrounds, background_crop_mode, rotationX, rotationY, rotationZ, margin, use_ligatures, augmenter, draw_contour, draw_contour_text) 575 permitted_contour=permitted_contour, 576 color=text_color, --> 577 draw_contour=draw_contour_text) 578 alpha = text_image[..., -1:].astype('float32') / 255 579 image = (alpha * text_image[..., :3] + (1 - alpha) * current_background).astype('uint8')

/usr/local/lib/python3.7/dist-packages/keras_ocr/data_generation.py in draw_text_image(text, fontsize, height, width, fonts, use_ligatures, thetaX, thetaY, thetaZ, color, permitted_contour, draw_contour) 354 fontsize=max(min_character_size, 1), 355 M=M, --> 356 contour=permitted_contour) 357 start_x = transformed_contour[:, 0].min() 358 start_y = transformed_contour[:, 1].min()`

` /usr/local/lib/python3.7/dist-packages/keras_ocr/data_generation.py in compute_transformed_contour(width, height, fontsize, M, contour, minarea) 466 x, y = slots_filtered[0][0] 467 contour = newContours[next( --> 468 index for index, contour in enumerate(newContours) 469 if cv2.pointPolygonTest(contour=contour, pt=(x, y), measureDist=False) >= 0)][:, 0, :] 470 return contour

/usr/local/lib/python3.7/dist-packages/keras_ocr/data_generation.py in (.0) 467 contour = newContours[next( 468 index for index, contour in enumerate(newContours) --> 469 if cv2.pointPolygonTest(contour=contour, pt=(x, y), measureDist=False) >= 0)][:, 0, :] 470 return contour 471

error: OpenCV(4.5.2) :-1: error: (-5:Bad argument) in function 'pointPolygonTest'

Overload resolution failed:

• Can't parse 'pt'. Sequence item with index 0 has a wrong type
• Can't parse 'pt'. Sequence item with index 0 has a wrong type`

1. Can you please tell me which version of Tensorflow and OpenCV being used since that might be the issue ?
2. I haven't changed any line of codes. Just running and going through your colab notebook. Please can you help ?
3. Have added new alphabets. Mostly added new punctuation which required for our custom model.

[LFBJC]

I experienced something similar and when I debugged the code I found out that it had to do with the dimensions and the margin, for example in the example you are trying the height is 100 and the margin is 50 when the code does image = image[margin:-margin, margin:-margin] the image becomes of size 0 I found several things to be improved on the example given for complete end-to-end training: the rotateX and rotateY arguments in the keras_ocr.data_generation.get_image_generator function are passed as floats when they are defined as ints on it's signature; There was a tensor shape mismatch and I needed to change the height and width of the image generator (I don't know much about this error as I didn't investigate it deeply, but it had something to do with an image croping the line where the exception happened had something like "image[:crop.shape[0],:crop.shape[1]] = crop") The callbacks raised exceptions and I needed to eliminate 2 of them (I don't know if this one is something on this repository or on keras); often there are exceptions due to some error on the zip extraction (also this I don't know if has to do with the zipfile library or this repository at the moment I'm struggling with this error).

[AdamSanderson93]

Hey all, not overly experienced with git so pardon the atrocious formatting but I fixed this by modifying the following....

Line 439 of data_generation.py old: for pt in [(x, y), (x2, y), (x2, y2), (x, y2)] new: for pt in [(int(x), int(y)), (int(x2), int(y)), (int(x2), int(y2)), (int(x), int(y2))]

Line 569 of data_generation.py old: if cv2.pointPolygonTest(contour=contour, pt=(x, y), measureDist=False) >= 0 new: if cv2.pointPolygonTest(contour=contour, pt=(int(x), int(y)), measureDist=False) >= 0

Not sure if this broke anything but I made it past the error and am training now.

[fncokg]

Hey all, not overly experienced with git so pardon the atrocious formatting but I fixed this by modifying the following....

Line 439 of data_generation.py old: for pt in [(x, y), (x2, y), (x2, y2), (x, y2)] new: for pt in [(int(x), int(y)), (int(x2), int(y)), (int(x2), int(y2)), (int(x), int(y2))]

Line 569 of data_generation.py old: if cv2.pointPolygonTest(contour=contour, pt=(x, y), measureDist=False) >= 0 new: if cv2.pointPolygonTest(contour=contour, pt=(int(x), int(y)), measureDist=False) >= 0

Not sure if this broke anything but I made it past the error and am training now.

That worked for me on Python3.9 and OpenCV4.7. Very much thanks!

But since this is a very simple bug to fix, and half a year passed, why the bug has not been fixed yet?