icinga2 icon indicating copy to clipboard operation
icinga2 copied to clipboard

Don't unnecessarily shuffle items before config validation

Open Al2Klimov opened this issue 1 year ago • 0 comments

Before ae693cb7e1df1b885142854cf8a0f8a7600a3fb7 (#9577) we've repeatedly looped over all items in parallel like this:

while not types.done:
  for t in types:
    if not t.done and t.dependencies.done:
      with parallel(all_items, CONCURRENCY) as some_items:
        for i in some_items:
          if i.type is t:
            i.commit()

I.e. all items got distributed over CONCURRENCY threads, but not always equally. E.g. it was the hosts' turn, but only two threads got hosts and did all the work. The others didn't do actual work (due to the lack of hosts in their queue) which reduced the performance. c721c302cd9c96bee25a20b3862dad347345648a (#6581) fixed it by shuffling all_items first. ae693cb7e1df1b885142854cf8a0f8a7600a3fb7 (#9577) made the latter unnecessary by replacing the above algorithm with this:

while not types.done:
  for t in types:
    if not t.done and t.dependencies.done:
      with parallel(all_items[t], CONCURRENCY) as some_items:
        for i in some_items:
          i.commit()

I.e. parallel() gets only items of type t, so all threads get e.g. hosts.

Al2Klimov avatar Feb 19 '24 13:02 Al2Klimov