SolarPanelDataWrangler icon indicating copy to clipboard operation
SolarPanelDataWrangler copied to clipboard

Optimize run_inference preprocessing/inference pipeline

Open typicalTYLER opened this issue 6 years ago • 1 comments

Currently this is the slowest part of the process, taking up the majority of the time per tile. However CPU utilization bounces around and never reaches 100% for all cores, and gpu utilization goes between 30% and 1%.

A couple idea for areas of improvement:

  • currently only runs one image in each inference batch, possible speedup if using bigger batches
  • no multiprocessing for image resizing
  • feed_dict used for inference (I've read this is not optimal here)

typicalTYLER avatar Apr 01 '19 18:04 typicalTYLER

RE multiprocessing for image querying/stitching/resizing, I was under the impression that sqlite was completely non concurrent, but apparently it is to some extent, I think enough to where we could create separate worker threads for querying and splitting, stitching, and resizing. We just might have to change the sqlite timeout setting, and verify that it's safe to do with sqlalchemy.

In terms of simplicity, the easiest process to write would probably be querying images and splitting, as it can be done in advance without too much change to the code. Since this step starts with querying slippy_tiles that don't have images yet, and ends with saving them to the disk and updating slippy_tiles.

It might be a little harder to come up with a parallel solution to the next steps in the preprocessing, stitching and resizing, as it's all done in memory and doesn't have the disk as a checkpoint.

typicalTYLER avatar Apr 10 '19 16:04 typicalTYLER