Error when using my own image.

[himansh123]

I tried running the short_test.ipynb, It works perfectly. But when I give my own image as input, It throws an Opencv error as below:

Error:

---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
<ipython-input-5-b5b2a510b083> in <module>()
     15     plt.show()
     16     correct = model.filename2formula(files[i])
---> 17     latex = model.predict(formula)
     18     print("Seq: ", latex['equation'])

G:\rahul sir\HE2LaTeX-master\Latex\Latex.py in predict(self, formula)
    296     def predict(self, formula):
    297         self.formula = formula
--> 298         self.get_bounding_boxes()
    299         self.normalize()
    300 

G:\rahul sir\HE2LaTeX-master\Latex\Latex.py in get_bounding_boxes(self)
    251             plt.imshow(thresh, cmap="gray")
    252             plt.show()
--> 253         _, contours, _ = cv2.findContours(thresh, cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)
    254 
    255         bounding_boxes = []

error: OpenCV(3.4.4) C:\projects\opencv-python\opencv\modules\imgproc\src\contours.cpp:195: error: (-210:Unsupported format or combination of formats) [Start]FindContours supports only CV_8UC1 images when mode != CV_RETR_FLOODFILL otherwise supports CV_32SC1 images only in function 'cvStartFindContours_Impl'

Can you tell me what thing I am doing wrong? also I am using a jpg image as below. Is there an issue with image?

[Wikunia]

How does the threshold image look like?

[himansh123]

Its totally black.

Than I used cv2.cvtColor(self.formula, cv2.COLOR_BGR2GRAY) to convert it to Gray image. which showed the threshold image perfectly, but throwed error on this line:

    formula_rects = self.add_rectangles(self.formula, bounding_boxes)

[Wikunia]

Okay good to know. Can you post the specific error as well?

[himansh123]

Here is the error I get when I use: cv2.cvtColor(self.formula, cv2.COLOR_BGR2GRAY)

<class 'numpy.ndarray'> (205, 354)

---

ValueError                                Traceback (most recent call last)
<ipython-input-5-0925160070b7> in <module>()
     16     plt.show()
     17     correct = model.filename2formula(files[i])
---> 18     latex = model.predict(formula)
     19     print("Seq: ", latex['equation'])

G:\rahul sir\HE2LaTeX-master\Latex\Latex.py in predict(self, formula)
    300     def predict(self, formula):
    301         self.formula = formula
--> 302         self.get_bounding_boxes()
    303         self.normalize()
    304 

G:\rahul sir\HE2LaTeX-master\Latex\Latex.py in get_bounding_boxes(self)
    273                 id_c += 1
    274         bounding_boxes = sorted(bounding_boxes, key=lambda k: (k['xmin'], k['ymin']))
--> 275         formula_rects = self.add_rectangles(self.formula, bounding_boxes)
    276         if self.plotting:
    277             print("Start bounding boxes: ")

G:\rahul sir\HE2LaTeX-master\Latex\Latex.py in add_rectangles(self, img, bounding_boxes)
    126             xmin, xmax = bounding_box['xmin'], bounding_box['xmax']
    127             ymin, ymax = bounding_box['ymin'], bounding_box['ymax']
--> 128             img_color[ymin,xmin:xmax] = [255,0,0]
    129             img_color[ymax-1,xmin:xmax] = [255,0,0]
    130             img_color[ymin:ymax,xmin] = [255,0,0]

ValueError: could not broadcast input array from shape (3) into shape (94,9)

[Wikunia]

I run this code with your image:

formula = io.imread("formulas/github.jpg")
formula = cv2.cvtColor(formula, cv2.COLOR_BGR2GRAY)
plt.figure(figsize=(20,10))
plt.imshow(formula, cmap="gray")
plt.show()
latex = model.predict(formula)
print("Seq: ", latex['equation'])

and there is no error. But the sequence predicted is quite off. This project is only a starting point. One would have to dive deeper into it to see where the problem is.

[himansh123]

OK. I got my mistake. But than again as u said results are not much satisfying. Can you please share the steps on how can I retrain the model?

[Wikunia]

I run it with verbose=True which shows me the single predictions per bounding box. That looks promising. So the problem is the Sequence model. Which you can train in Seq2Seq.ipynb Which means you need to create your own sequences which I currently do in generate.ipynb. Here I have simple and fraction. Probably we need more randomness in both of them.

[TanayKarve]

Please check out this answer