awsweeper icon indicating copy to clipboard operation
awsweeper copied to clipboard

Discovery Performance

Open kadaan opened this issue 3 years ago • 3 comments

The performance of discovery is pretty slow as it is a serial process for all available resources.

The improvements I see are:

  1. Parallelize discovery by type
  2. Initial filtering of discovered resources. If all data for filtering exists on the resources then apply the filter and take the result. If all data for filtering is not available then include the result for later processing.
  3. Parallelize updating of resource state by using a worker pool which handles resources after 2.
  4. Final filtering of discovered resources, this time with state
  5. Collect all results from 4. and return them as the result from List()

kadaan avatar Jun 25 '22 20:06 kadaan

@jckuester Feedback?

kadaan avatar Jun 25 '22 20:06 kadaan

See: https://github.com/kadaan/awsweeper/tree/speed_up

kadaan avatar Jun 25 '22 20:06 kadaan

I'll have a look :slightly_smiling_face:

jckuester avatar Jun 27 '22 14:06 jckuester