FACT_core
FACT_core copied to clipboard
fkie-cad
speeding up analysis
Do you have any recommendations on speeding up analysis? I have a distributed setup with the backend running exclusively on its own server. I uploaded several firmware with the minimal set of analysis tasks and the backend isn't really seeing much utilization even though analysis has been going on for over 13 hours.
I bumped up the unpack thread count to 12, but maybe it should be increased more?
It seems to be still limited by unpacking: There is an additional progress bar displaying the unpacking progress (in yellow) and there seems to be no firmware with more analysis then unpacking progress. Are you running FACT on a HDD or SSD? File I/O could also slow down unpacking. Are there any error messages in the logs?
Please do mind that you need to restart FACT for the changed unpacking thread count to take effect and this would abort currently running analyses. It is generally advisable to not analyze too many firmware images in parallel because of how the scheduling works: Unpacked files from firmware are recursively added to the scheduling process and will be queued at the back, behind files from other firmware that may have been uploaded later (meaning it will take a long time before the first analysis is finished).
Improving unpacking performance is definitely on our agenda: since the extraction process runs in a docker container and we start a new container for each unpacked file, there is a lot of overhead for small files (which there are a lot of in this case). We are looking into changing this so that the unpacking containers are running constantly and use a task queue system for scheduling unpacking tasks but this is not something that will be completed in the close future.
If this:
It is generally advisable to not analyze too many firmware images in parallel because of how the scheduling works:
is true, is there a way to limit the number of parallel extractions are initiated at once? Maybe this could be in the main config file?
If this:
It is generally advisable to not analyze too many firmware images in parallel because of how the scheduling works:
is true, is there a way to limit the number of parallel extractions are initiated at once? Maybe this could be in the main config file?
Hm. I think this is misleading. @jstucke refers to this axiom because it generally leads to overrun memory. Though with enough memory there should not necessarily be a problem. You limit the number of parallel extractions with the unpacking threads, so @jstucke already said to possibly up that number. Not reduce it :sweat_smile:
When we do batch analyses, I usually script them using the REST API to start the next analysis once the previous is done. That way you will not encounter problems in 99 % of firmware samples. Generally this should not come with a notable performance dip compared to concurrent analyses. We could probably provide sample code for the REST upload part.
What I'm attempting to suggest is that instead of putting the responsibility on the upload script to process the firmware individually and check the number of currently processing in the queue, provide that functionality in the server itself so that it doesn't get overloaded.
I've had some issues in batch processing that were not ideal because when you have that many firmware images in the processing queue and something fails you have to start over from scratch for all the images that were not completely unpacked. This batch I processed above took well over 24 hours before the first one completed.
I agree that it would be nice to have this in FACT. Since there doesn't seem to be a "multiprocessing-safe" priority queue implementation in python, there seems to be no easy solution to this. We plan to rewrite our scheduling (maybe with RabbitMQ or something similar) at some point, though. Until then, an upload script would be the best solution.
You can get the status of FACT (including the info if an analysis is currently running) via the REST API.
Recent changes to the unpacking and analysis schedulers should improve scaling on systems with many cores a lot. I will close this issue since it hasn't been updated for a long time. Feel free to reopen it if you have additional questions.