[FEATURE]: Batching `.extract_faces`

[galthran-wq]

Description

Related: https://github.com/serengil/deepface/pull/1433 https://github.com/serengil/deepface/issues/1101

I think it would be a good idea to be able to batch input images on detection. Many currently existing detectors are natively able to to perform batched inference (to name a few, YOLO, retinaface, MTCNN) The hypothesis is that batching would improve the performance significantly. Partially validated by https://github.com/serengil/retinaface/pull/116. Since retinaface is not going to support batching, we could at least have YOLO, MTCNN and some others

@serengil what do you think?

Additional Info

No response

[NatLee]

Hello, I have a question regarding .extract_faces. If it supports batch input, would the subsequent models, such as emotion or age detection, also need to be adjusted based on the output of .extract_faces?

For instance, if I have three input images, each will produce face detection results. How should the subsequent analysis define multiple faces across these three images? Perhaps using the input order to each face could be a solution?

[galthran-wq]

hi. I do not think they would have to be adjusted, because the function is supposed to work exactly the same for a single image input and all the susequent models are using it currently with a single image.

Now, if we were to adjust the usage in subsequent models to batched inputs, then you indeed raise a good point on how to recover which detection results correspond to which images in the input. And I think the solution here is this. The output type of the .extract_faces should be Union[List[Dict[str, Any]], List[List[Dict[str, Any]]. So, it is List[Dict[str, Any]] for a single image input (just like before), and List[List[Dict[str, Any] if a list of images is passed -- for each input face, a list of detected faces.

I'm planing to incorporate this change in the PR

[serengil]

@galthran-wq so, if you feed 3 images to input as list or numpy and the 2nd one has 5 faces, then function will still return a list of 3. The second item in the response will have 5 items. Liked that design.

[chirag-agrawal24]

@serengil Recently I was working on a surveillance project. I used deepface for face recognition. I have trained a SVM model for classification of person using face representation given by ArcFace model. As deepface library does not support batch processing, I updated some code for representation.py , detection.py and Yolo.py for batch processing.

Now I see this issue and want to contribute to the project. I have attached the zip file(batch_deepface.zip) for those updated code. This week, I will commit the changes in the fork, would notify about the PR.

My approach:

• Input: List of image/img path ( for single image ,give input like: [img,])

• Output(representation): List[ List [Dict [str, Any]]]

  • where each sublist contains dictionaries for each detected face in the image.

• Detection model also work in this principle.

• For representation batching, I provide 3 approach:

  • Create batch of input for model (Also save the index of respective image) . Forward it. Using model output and image index, format the result .
  • Only batch detection , no batching for recognition
  • Create N batches for N images, where each batch contains detected faces from respective image. (No need for image index)

I would be thankful if you review my approach and code.

[serengil]

@chirag-agrawal24

• I cannot review changes from zip, still I can add some comments according to the comments you shared
• Input: List of image/img path ( for single image ,give input like: [img,]) -> for single image, you shouldn't send it like [img,], it shouldn't be changed
• Output: if the input list, output can be List[ List [Dict [str, Any]]] but if it is not list then it should be List [Dict [str, Any]]
• Representation and Analyze methods are already supporting bactches.

[chirag-agrawal24]

@serengil Thank you for your reply.

• I will consider your suggestion for input as it will be convenient for user.

• Should I also give an input parameter for BATCH_SIZE, device ("cuda","cpu") in those functions?

• Previously I was using a deepface version installed using pip(where I do not have batch detection in representation). Now I see the latest github code have batch detection. After seeing those code, I have some doubts:

  • Will it handle the case, where user give a lot of images, or hundreds of faces are detected in those images.
  • Will it good if we first create batches of input with user defined batch size and pass it to model batch by batch OR just create a single batch for all input and forward it.