execnet icon indicating copy to clipboard operation
execnet copied to clipboard

Group.__init() should be parallelised

Open pytestbot opened this issue 8 years ago • 1 comments

  • Bitbucket: https://bitbucket.org/hpk42/execnet/issue/28
  • Originally reported by: @crusaderky
  • Originally created at: 2014-01-22T16:51:21.959

Consider the following snippet:

hosts = [...list of hosts...]
group = execnet.Group(("ssh %s" % host for host in hosts))
group.remote_exec(something)

Imagine having a grid of 200 hosts. While Group.remote_exec() will run on all the hosts in parallel, Group.__init__() will connect via ssh to each one of them, serially. On a typical LAN, this easily accounts to minutes.

I attach a patch for multi.py (on top of the latest hg commit) that is intended to fix the problem. However, with that patch the program deadlocks in gateway_base.py, line 360, and I can't figure out the reason.

pytestbot avatar Jul 21 '17 10:07 pytestbot

Original comment by @RonnyPfannschmidt

the patch is a good idea, but we need to take the new event handlers into account, might be a item for version 1.4

pytestbot avatar Jul 21 '17 10:07 pytestbot