how to generate training_data.pickle?

[mylovelycodes]

thanks!

[kavishdahekar]

Sorry, forgot to mention that in the README. You can either use the one I have already generated here : master/training/training_data.pickle or follow the instructions below.

First of all training_data.pickle simply contains a dictionary. The keys of the dictionary are all possible characters observed in the captchas like 'a','b',.. etc. There are a total of 28 such keys. Under each key is a python list where we will add data of all the different images we found of that character from the training data.

To generate this dictionary, read all folders from /master/data_preparation/segmented_data. The folder's name will become the key and all the images in that folder will be the data under that key. Initialize an empty list under the key. Read all the images in the folder. Each image is 44x65, transform it into a numpy array of size 1x2925(row vector made from appending all 44 rows of the image, and one extra row filled with zeroes) containing binary values. The value will be 0 if the pixel in the image was black and 1 if the pixel in the image was white. Append these vectors to the list. (You can find code for doing all of this in solveCaptcha.py from line 124 to 131) Final dictionary will look something like this:

a: [
       array([[0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
       array([[0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
       array([[0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
       ..,
       so on for all images of 'a',
       ..
    ]
b: [
       array([[0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
       array([[0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
       array([[0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
       ..,
       so on for all images of 'b',
       ..
    ]
.. so on for all characters ...

Store the dictionary to training_data.pickle.

Hope that helped. Let me know if you have any more questions.

[franz101]

Could you provide a code generating the pickle file? 🙏

[kavishdahekar]

Sure. Code (pseudo?) below:

# create empty dictionary
training_data = {}

loop over each of the folders in data_preparation/segmented_data : curFolderName
# these folders and files are generated in segregate.py (line 126). The folder name is the alphabet that is stored in the pngs in that folder.
    loop over each file in curFolderName : curFile.png
        curImage = cv2.imread("curFile.png",0) # second argument specifies we are reading the image in grayscale
        # we flatten the image into a row vector
        curImageFlat = curImage.flatten()
        # each pixel in this image is either 0 (black) or 255(white) (done in segregate.py line 46)
        # we will convert the 255's to 1's
        curImageFlat[curImageFlat == 255] = 1
        # convert to uint8 (this will be converted to np.float32 before feeding to tensorflow later but you can do it here itself too)
        curImageVector = curImageFlat.astype(np.uint8)

        # add to training_data dictionary
        if curFolderName not in training_data:
            training_data[curFolderName] = []
        training_data[curFolderName].append(curImageVector)

# finally write the training_data dictionary to a pickle file
pickle.dump( training_data , open("training_data.pickle","wb") )

I should have added and explained this in the repository itself, completely escaped my mind. Apologies. Nonetheless, hope the above code helps(wrote it directly in the comment box, so there might be some syntax/logic errors).

If you are able to successfully generate training_data.pickle please consider raising a PR.