appending functionality to .pkl files in deepface find

[LOANPIA]

Dear Mr.Sefik! I like the combination between facenet and mtcnn for face recognition, not analyze But I have 2 troubles: 1/ When I execute DeepFace.find, this function generates a file in folder (representations_facenet.pkl), I want to append for new value, new image, I failed because I don't know format statement in file
df = pd.DataFrame(representations, columns = ["identity", "%s_representation" % (model_name)]). How to do that? I tried to write file [a+] or [wb] and that thing, it doesn't work with next time detection | recognition 2/ the reason I do the number 1 thing is - Loading speed for 1 vector file takes a lot of seconds each time ~43 seconds of Facenet model, I agree that the second time is very fast, only 8 seconds [time import, time find function], but the recognition is not correct. Finally, I have to delete the old pkl file and do again. There is no way to improve adding new to the pkl . file. I want to add or combine file, and speed more faster, about 3-5 seconds.

 Thats all | Help me, thanks

[LOANPIA]

What I said above is similar to this issue but I don't understand your answer, your answer doesn't provide where I want to search

[serengil]

this is a pickle file. you cannot append a line into it. you firstly need to read it, store its content to pandas dataframe, then save as pickle as well. i recommend you to search python pickle module.

current design is based on deleting the pickle. i can add an append functionality in the next release but i might not do it soon.

[serengil]

i rename the issue name. this will be open until the next relase.

[LOANPIA]

Yeah, and you could told me time release for sure btw, I want to receive the reply in option 2 - improve time, I performed with load model in it but delay time still 8 seconds, do you think image need to be high resolution? What is the standard value? I get used to do sharpness all images

[serengil]

all images are resized to (160, 160, 3) for facenet after face detected and aligned. so, even if you feed a high resolution image, then it is going downscaled.

[LOANPIA]

What do you think about Pytorch could be append new object, new embedding into (.pth) file, Can you help me update next release to convert this issue in (.pkl) like above discuss?

[LOANPIA]

hello, I'm back - have you solved it yet?

[serengil]

as i mentioned, i cannot do it soon. meanwhile, this issue will stay in open status.

[BeatRoy]

I can help if I know what needs to be changed @LOANPIA. What should be the solution to your problem?

[LOANPIA]

I can help if I know what needs to be changed @LOANPIA. What should be the solution to your problem?

ok @BeatRoy, well - in this deepface frame, I would like to know if there is a way to write at the end of the file (.pkl) that was first trained to improve recognition time I used "a+" variable, always happen binary error I try "wb" but not true because I don't know format in pickle trained file generated

[BeatRoy]

I can help if I know what needs to be changed @LOANPIA. What should be the solution to your problem?

ok @BeatRoy, well - in this deepface frame, I would like to know if there is a way to write at the end of the file (.pkl) that was first trained to improve recognition time I used "a+" variable, always happen binary error I try "wb" but not true because I don't know format in pickle trained file generated

How I append to pickle files is read them and append faces to the array and than save them back to the pickle file like so:

# representations needs to be the array with the new trained faces
with open("picklefile.pkl"), 'rb') as f:
    representations += pickle.load(f)

with open("picklefile.pkl"), 'wb' as f:
    pickle.dump(representations, f)

I hope this solves your problem

[LOANPIA]

I can help if I know what needs to be changed @LOANPIA. What should be the solution to your problem?

ok @BeatRoy, well - in this deepface frame, I would like to know if there is a way to write at the end of the file (.pkl) that was first trained to improve recognition time I used "a+" variable, always happen binary error I try "wb" but not true because I don't know format in pickle trained file generated

How I append to pickle files is read them and append faces to the array and than save them back to the pickle file like so:

# representations needs to be the array with the new trained faces
with open("picklefile.pkl"), 'rb') as f:
    representations += pickle.load(f)

with open("picklefile.pkl"), 'wb' as f:
    pickle.dump(representations, f)

I hope this solves your problem

Thank you for all, as I mentioned I couldn't append like encoding similar pkl file model trained: Like this -- This picture show 128 features, I think so~ But I didn't find out the format, and then I appended by "wb", It generated the pkl file such as result line: Your code, I tried and it just is that >> btw,Can I ask you about "representations" key represents what?

Finally, I hope to find a demonstration repo about the pickle file that the model trained

[LOANPIA]

This is Find Function I used, have some information here I realize that I keep pickle file with default some images trained, new ones have to delete only I am waiting for new release

[OttomanZ]

Okay, so for the community, here is the answer on how we can go ahead and append representations to the .pkl file in automatically generated in our database folder.

Reading the Saved Encodings from the representations.pkl

import pandas as pd
output = list(pd.read_pickle('./database/representations_vgg_.pkl'))
os.remove('./database/representations_vgg_.pkl')

Appending and Saving new representations

So, if you go ahead and look closely it follows the following syntax, @LOANPIA

[
    ["./database/path_to_your_image.jpg",
        [ face_encoding  ]
    ],
]

So, you can then go ahead and save the encodings in the python file, this will go ahead and save you a ton of time, when you are live adding new entries into the database, as otherwise when you go ahead and delete representations.pkl it goes ahead and re-encodes, everything (valuable time in a time-focused environment). That is good in some situations @serengil (Don't Depreciate this Design Choice), and I hope that this clears this issue for any further people looking to solve the following problem.

face encoding? you may ask.

To go ahead and get the face encoding, you need to go ahead and use the following function from the deepface.DeepFace class. You may also find this in the docs.

from deepface import DeepFace
DeepFace.represent(img_path = "img.jpg")

Hope this Helps!

[OttomanZ]

@serengil You can go ahead and close this one now, as I can confirm my approach works. Also, I wanted to say thank you for the awesome work you are doing. I'm quite thankful to you for contributing so much to the open source community. Sefik, Thanks this has helped out tremendously in some of my recent projects.

sudo echo "thanks"