dask-ec2 dask_ec2.libpepper.PepperException: Server error.

dask_ec2.libpepper.PepperException: Server error.

Open lionfish0 opened this issue 7 years ago • 1 comments

I get this exception when "installing workers". It doesn't happen all the time. I wonder if there's a race condition or similar that's causing it? Maybe something on the servers isn't initialised in time? Command: dask-ec2 up --keyname research --keypair .ssh/research.pem --region-name eu-west-1 --ami ami-c8b51fb1 --tags research:dp --count 1 --spot-count 16 --spot-price 0.796 --volume-size 10 --type c4.large --nprocs 2 I should say that I'm using my forked version of this repo.

The error messages:

Installing scheduler
+---------+----------------------+-----------------+
| Node ID | # Successful actions | # Failed action |
+=========+======================+=================+
| node-0  | 22                   | 0               |
+---------+----------------------+-----------------+
Installing workers
Traceback (most recent call last):
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/libpepper.py", line 130, in req
    f = urlopen(req, context=con)
  File "/home/mike/anaconda3/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/home/mike/anaconda3/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/home/mike/anaconda3/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/home/mike/anaconda3/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/home/mike/anaconda3/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/home/mike/anaconda3/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 500: Internal Server Error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/cli/main.py", line 23, in start
    cli(obj={})
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/cli/main.py", line 201, in up
    dask=dask, notebook=notebook, nprocs=nprocs, source=source)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/cli/main.py", line 347, in provision
    ctx.invoke(dask_install, filepath=filepath, nprocs=nprocs, source=source)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/cli/daskd.py", line 80, in dask_install
    output = cluster.salt_call("node-[1-9]*", "state.sls", ["dask.distributed.worker"])
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/cluster.py", line 65, in salt_call
    return self.pepper.local(target, module, args)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/libpepper.py", line 225, in local
    return self.low([low], path='/')
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/libpepper.py", line 200, in low
    return self.req(path, lowstate)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/libpepper.py", line 142, in req
    raise PepperException('Server error.')
dask_ec2.libpepper.PepperException: Server error.

The whole output:

DEBUG: Searching for default VPC DEBUG: Default VPC found - Using VPC ID: vpc-7f49ce1a DEBUG: Searching for default subnet in VPC vpc-7f49ce1a DEBUG: Default subnet found - Using Subnet ID: subnet-8a0199fd Launching nodes DEBUG: Checking that keyname 'research' exists on EC2 DEBUG: Checking that security group 'dask-ec2-default' exists on EC2 DEBUG: Getting all security groups and filtering by VPC ID vpc-7f49ce1a and name dask-ec2-default DEBUG: Found Security groups: [ec2.SecurityGroup(id='sg-9146afe9')] DEBUG: Getting all security groups and filtering by VPC ID vpc-7f49ce1a and name dask-ec2-default DEBUG: Found Security groups: [ec2.SecurityGroup(id='sg-9146afe9')] DEBUG: Creating 16 spot instances on EC2 DEBUG: Creating 1 instances on EC2 DEBUG: spot request `sir-4xj866en` fulfilled! DEBUG: spot request `sir-idh86iwm` fulfilled! DEBUG: spot request `sir-aeti6fsq` fulfilled! DEBUG: spot request `sir-i9c85n5p` fulfilled! DEBUG: spot request `sir-wpvr5jnn` fulfilled! DEBUG: spot request `sir-65t87h7q` fulfilled! DEBUG: spot request `sir-e2mr463m` fulfilled! DEBUG: spot request `sir-5f2i5hpq` fulfilled! DEBUG: spot request `sir-txzg6kjm` fulfilled! DEBUG: spot request `sir-32pr4app` fulfilled! DEBUG: spot request `sir-p8e87jip` fulfilled! DEBUG: spot request `sir-ppwi4fqm` fulfilled! DEBUG: spot request `sir-eg986wwn` fulfilled! DEBUG: spot request `sir-ctz84wtq` fulfilled! DEBUG: spot request `sir-mntr5gam` fulfilled! DEBUG: spot request `sir-t79g65ap` fulfilled! DEBUG: all spots fulfilled! DEBUG: Tagging instance 'i-0e1bddcde99715e6c' DEBUG: Tagging instance 'i-0c96a28a6933f073c' DEBUG: Tagging instance 'i-0c980a3ed3edb5078' DEBUG: Tagging instance 'i-0618b6ae501690189' DEBUG: Tagging instance 'i-0221d206dcfb9a982' DEBUG: Tagging instance 'i-064379cd368ff4288' DEBUG: Tagging instance 'i-0333b5e5f67e7728b' DEBUG: Tagging instance 'i-0b3265f862d4b662b' DEBUG: Tagging instance 'i-0f6edef96ac796c49' DEBUG: Tagging instance 'i-0106523d86c02f84f' DEBUG: Tagging instance 'i-0a1217026dd6f511e' DEBUG: Tagging instance 'i-0a9280704174aa0dc' DEBUG: Tagging instance 'i-06ec5d5d579372d53' DEBUG: Tagging instance 'i-0aaab406064d9d47e' DEBUG: Tagging instance 'i-04ae4cee6255f70fd' DEBUG: Tagging instance 'i-0e4e84e330da6c5f9' DEBUG: Tagging instance 'i-05692d9ccabeba9a1' Checking SSH connection to nodes DEBUG: Checking ssh connection for 34.253.212.200 DEBUG: Running command bash -c 'ls' on '34.253.212.200' DEBUG: Checking ssh connection for 54.154.40.56 DEBUG: Running command bash -c 'ls' on '54.154.40.56' DEBUG: Checking ssh connection for 34.242.214.91 DEBUG: Running command bash -c 'ls' on '34.242.214.91' DEBUG: Checking ssh connection for 54.171.53.67 DEBUG: Running command bash -c 'ls' on '54.171.53.67' DEBUG: Checking ssh connection for 54.171.131.4 DEBUG: Running command bash -c 'ls' on '54.171.131.4' DEBUG: Checking ssh connection for 54.229.249.113 DEBUG: Running command bash -c 'ls' on '54.229.249.113' DEBUG: Checking ssh connection for 34.253.208.68 DEBUG: Running command bash -c 'ls' on '34.253.208.68' DEBUG: Checking ssh connection for 54.171.211.136 DEBUG: Running command bash -c 'ls' on '54.171.211.136' DEBUG: Checking ssh connection for 54.229.24.59 DEBUG: Running command bash -c 'ls' on '54.229.24.59' DEBUG: Checking ssh connection for 54.229.55.142 DEBUG: Running command bash -c 'ls' on '54.229.55.142' DEBUG: Checking ssh connection for 54.171.43.34 DEBUG: Running command bash -c 'ls' on '54.171.43.34' DEBUG: Checking ssh connection for 54.194.52.177 DEBUG: Running command bash -c 'ls' on '54.194.52.177' DEBUG: Checking ssh connection for 54.246.136.108 DEBUG: Running command bash -c 'ls' on '54.246.136.108' DEBUG: Checking ssh connection for 54.171.209.246 DEBUG: Running command bash -c 'ls' on '54.171.209.246' DEBUG: Checking ssh connection for 54.246.229.87 DEBUG: Running command bash -c 'ls' on '54.246.229.87' DEBUG: Checking ssh connection for 34.242.164.173 DEBUG: Running command bash -c 'ls' on '34.242.164.173' DEBUG: Checking ssh connection for 176.34.137.27 DEBUG: Running command bash -c 'ls' on '176.34.137.27' +-------------------+-----------+ | Node IP | SSH check | +===================+===========+ | 34.253.212.200:22 | True | | 54.154.40.56:22 | True | | 34.242.214.91:22 | True | | 54.171.53.67:22 | True | | 54.171.131.4:22 | True | | 54.229.249.113:22 | True | | 34.253.208.68:22 | True | | 54.171.211.136:22 | True | | 54.229.24.59:22 | True | | 54.229.55.142:22 | True | | 54.171.43.34:22 | True | | 54.194.52.177:22 | True | | 54.246.136.108:22 | True | | 54.171.209.246:22 | True | | 54.246.229.87:22 | True | | 34.242.164.173:22 | True | | 176.34.137.27:22 | True | +-------------------+-----------+ Bootstrapping salt master DEBUG: Running command sudo -S bash -c 'curl -sS -L https://bootstrap.saltstack.com | sh -s -- -d -X -M -N stable' on '34.253.212.200' DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -M -N -P -L -p salt-api stable' on '34.253.212.200' DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/auto_accept.conf to /tmp/.__tmp_copy DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/master.d/auto_accept.conf' on '34.253.212.200' DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.253.212.200' DEBUG: Running command sudo -S bash -c 'apt-get install -y python-pip libssl-dev libffi-dev python-dev' on '34.253.212.200' DEBUG: Running command sudo -S bash -c 'pip install --upgrade pip packaging appdirs six' on '34.253.212.200' DEBUG: Running command sudo -S bash -c 'pip install cherrypy==3.2.3' on '34.253.212.200' DEBUG: Running command sudo -S bash -c 'pip install PyOpenSSL==16.2.0' on '34.253.212.200' DEBUG: Running command sudo -S bash -c 'salt-call --local tls.create_self_signed_cert' on '34.253.212.200' DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/rest_cherrypy.conf to /tmp/.__tmp_copy DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/master.d/rest_cherrypy.conf' on '34.253.212.200' DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.253.212.200' DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/external_auth.conf to /tmp/.__tmp_copy DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/master.d/external_auth.conf' on '34.253.212.200' DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.253.212.200' DEBUG: Running command sudo -S bash -c 'id -u saltdev &>/dev/null || useradd -p $(openssl passwd -1 saltdev) saltdev' on '34.253.212.200' DEBUG: Running command sudo -S bash -c 'service salt-master restart' on '34.253.212.200' DEBUG: Running command sudo -S bash -c 'service salt-api restart' on '34.253.212.200' Bootstrapping salt minions DEBUG: Installing salt-minion on all the nodes DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-6 stable' on '34.253.208.68' DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-12 stable' on '54.246.136.108' DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-3 stable' on '54.171.53.67' DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-2 stable' on '34.242.214.91' DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-1 stable' on '54.154.40.56' DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-5 stable' on '54.229.249.113' DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-7 stable' on '54.171.211.136' DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-13 stable' on '54.171.209.246' DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-9 stable' on '54.229.55.142' DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-8 stable' on '54.229.24.59' DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-4 stable' on '54.171.131.4' DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-10 stable' on '54.171.43.34' DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-14 stable' on '54.246.229.87' DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-15 stable' on '34.242.164.173' DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-11 stable' on '54.194.52.177' DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-16 stable' on '176.34.137.27' DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-0 stable' on '34.253.212.200' DEBUG: Configuring salt-mine on the salt minions DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '54.229.24.59' DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '34.253.212.200' DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '54.229.55.142' DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '54.171.53.67' DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '34.242.214.91' DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '54.229.249.113' DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '54.154.40.56' DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '54.246.229.87' DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '54.246.136.108' DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '176.34.137.27' DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '34.242.164.173' DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '54.171.209.246' DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '54.171.131.4' DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '34.253.208.68' DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '54.171.43.34' DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '54.171.211.136' DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '54.194.52.177' DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '54.229.24.59' DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '54.229.55.142' DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '54.171.53.67' DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.253.212.200' DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.242.214.91' DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '54.154.40.56' DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '54.246.229.87' DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '54.229.249.113' DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.242.164.173' DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '176.34.137.27' DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '54.171.131.4' DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.253.208.68' DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '54.171.209.246' DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '54.246.136.108' DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '54.171.43.34' DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '54.171.211.136' DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '54.194.52.177' DEBUG: Restarting the salt-minion service DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '54.171.131.4' DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '54.171.211.136' DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '54.229.55.142' DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '54.171.53.67' DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '54.229.249.113' DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '54.171.43.34' DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '34.253.208.68' DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '54.171.209.246' DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '54.229.24.59' DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '54.154.40.56' DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '54.194.52.177' DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '34.253.212.200' DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '34.242.214.91' DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '34.242.164.173' DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '176.34.137.27' DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '54.246.229.87' DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '54.246.136.108' Uploading salt formulas DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt to /srv/salt DEBUG: Running command sudo -S bash -c 'mkdir -p /srv/salt' on '34.253.212.200' DEBUG: Creating directory /tmp/.__tmp_copy mode=511 DEBUG: Creating directory /tmp/.__tmp_copy/supervisor mode=511 DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/supervisor to /tmp/.__tmp_copy/supervisor DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/supervisor' on '34.253.212.200' DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/supervisor/init.sls to /tmp/.__tmp_copy/supervisor/init.sls DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/supervisor/settings.sls to /tmp/.__tmp_copy/supervisor/settings.sls DEBUG: Creating directory /tmp/.__tmp_copy/dask mode=511 DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask to /tmp/.__tmp_copy/dask DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/dask' on '34.253.212.200' DEBUG: Creating directory /tmp/.__tmp_copy/dask/distributed mode=511 DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed to /tmp/.__tmp_copy/dask/distributed DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/dask/distributed' on '34.253.212.200' DEBUG: Creating directory /tmp/.__tmp_copy/dask/distributed/worker mode=511 DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/worker to /tmp/.__tmp_copy/dask/distributed/worker DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/dask/distributed/worker' on '34.253.212.200' DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/worker/init.sls to /tmp/.__tmp_copy/dask/distributed/worker/init.sls DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/debug.sls to /tmp/.__tmp_copy/dask/distributed/debug.sls DEBUG: Creating directory /tmp/.__tmp_copy/dask/distributed/scheduler mode=511 DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/scheduler to /tmp/.__tmp_copy/dask/distributed/scheduler DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/dask/distributed/scheduler' on '34.253.212.200' DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/scheduler/init.sls to /tmp/.__tmp_copy/dask/distributed/scheduler/init.sls DEBUG: Creating directory /tmp/.__tmp_copy/dask/distributed/templates mode=511 DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/templates to /tmp/.__tmp_copy/dask/distributed/templates DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/dask/distributed/templates' on '34.253.212.200' DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/templates/dask-worker.conf to /tmp/.__tmp_copy/dask/distributed/templates/dask-worker.conf DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/templates/dask-scheduler.conf to /tmp/.__tmp_copy/dask/distributed/templates/dask-scheduler.conf DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/init.sls to /tmp/.__tmp_copy/dask/distributed/init.sls DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/settings.sls to /tmp/.__tmp_copy/dask/distributed/settings.sls DEBUG: Creating directory /tmp/.__tmp_copy/conda mode=511 DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/conda to /tmp/.__tmp_copy/conda DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/conda' on '34.253.212.200' DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/conda/debug.sls to /tmp/.__tmp_copy/conda/debug.sls DEBUG: Creating directory /tmp/.__tmp_copy/conda/templates mode=511 DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/conda/templates to /tmp/.__tmp_copy/conda/templates DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/conda/templates' on '34.253.212.200' DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/conda/templates/conda.sh to /tmp/.__tmp_copy/conda/templates/conda.sh DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/conda/init.sls to /tmp/.__tmp_copy/conda/init.sls DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/conda/settings.sls to /tmp/.__tmp_copy/conda/settings.sls DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/top.sls to /tmp/.__tmp_copy/top.sls DEBUG: Creating directory /tmp/.__tmp_copy/jupyter mode=511 DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter to /tmp/.__tmp_copy/jupyter DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/jupyter' on '34.253.212.200' DEBUG: Creating directory /tmp/.__tmp_copy/jupyter/templates mode=511 DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter/templates to /tmp/.__tmp_copy/jupyter/templates DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/jupyter/templates' on '34.253.212.200' DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter/templates/jupyter_notebook_config_py to /tmp/.__tmp_copy/jupyter/templates/jupyter_notebook_config_py DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter/templates/jupyter-notebook.conf to /tmp/.__tmp_copy/jupyter/templates/jupyter-notebook.conf DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter/init.sls to /tmp/.__tmp_copy/jupyter/init.sls DEBUG: Creating directory /tmp/.__tmp_copy/jupyter/notebook mode=511 DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter/notebook to /tmp/.__tmp_copy/jupyter/notebook DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/jupyter/notebook' on '34.253.212.200' DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter/notebook/init.sls to /tmp/.__tmp_copy/jupyter/notebook/init.sls DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter/notebook/settings.sls to /tmp/.__tmp_copy/jupyter/notebook/settings.sls DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter/settings.sls to /tmp/.__tmp_copy/jupyter/settings.sls DEBUG: Creating directory /tmp/.__tmp_copy/java mode=511 DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/java to /tmp/.__tmp_copy/java DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/java' on '34.253.212.200' DEBUG: Creating directory /tmp/.__tmp_copy/java/openjdk mode=511 DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/openjdk to /tmp/.__tmp_copy/java/openjdk DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/java/openjdk' on '34.253.212.200' DEBUG: Creating directory /tmp/.__tmp_copy/java/openjdk/templates mode=511 DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/openjdk/templates to /tmp/.__tmp_copy/java/openjdk/templates DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/java/openjdk/templates' on '34.253.212.200' DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/openjdk/templates/java.sh to /tmp/.__tmp_copy/java/openjdk/templates/java.sh DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/openjdk/init.sls to /tmp/.__tmp_copy/java/openjdk/init.sls DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/openjdk/env.sls to /tmp/.__tmp_copy/java/openjdk/env.sls DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/openjdk/settings.sls to /tmp/.__tmp_copy/java/openjdk/settings.sls DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/debug.sls to /tmp/.__tmp_copy/java/debug.sls DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/init.sls to /tmp/.__tmp_copy/java/init.sls DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/settings.sls to /tmp/.__tmp_copy/java/settings.sls DEBUG: Creating directory /tmp/.__tmp_copy/system mode=511 DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/system to /tmp/.__tmp_copy/system DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/system' on '34.253.212.200' DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/system/macros.sls to /tmp/.__tmp_copy/system/macros.sls DEBUG: Creating directory /tmp/.__tmp_copy/system/templates mode=511 DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/system/templates to /tmp/.__tmp_copy/system/templates DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/system/templates' on '34.253.212.200' DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/system/templates/limits.conf to /tmp/.__tmp_copy/system/templates/limits.conf DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/system/base.sls to /tmp/.__tmp_copy/system/base.sls DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy/* /srv/salt' on '34.253.212.200' DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.253.212.200' DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/pillar to /srv/pillar DEBUG: Running command sudo -S bash -c 'mkdir -p /srv/pillar' on '34.253.212.200' DEBUG: Creating directory /tmp/.__tmp_copy mode=511 DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/pillar/top.sls to /tmp/.__tmp_copy/top.sls DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy/* /srv/pillar' on '34.253.212.200' DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.253.212.200' Uploading conda and cluster settings DEBUG: Uploading file /tmp/tmpgyvuu5cf to /tmp/.__tmp_copy DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /srv/pillar/conda.sls' on '34.253.212.200' DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.253.212.200' DEBUG: Uploading file /tmp/tmpe6af0crt to /tmp/.__tmp_copy DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /srv/pillar/cluster.sls' on '34.253.212.200' DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.253.212.200' +---------+----------------------+-----------------+ | Node ID | # Successful actions | # Failed action | +=========+======================+=================+ | node-11 | 6 | 0 | +---------+----------------------+-----------------+ DEBUG: Uploading file /tmp/tmpi_4r_utk to /tmp/.__tmp_copy DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /srv/pillar/dask.sls' on '34.253.212.200' DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.253.212.200' Installing scheduler +---------+----------------------+-----------------+ | Node ID | # Successful actions | # Failed action | +=========+======================+=================+ | node-0 | 22 | 0 | +---------+----------------------+-----------------+ Installing workers Traceback (most recent call last): File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/libpepper.py", line 130, in req f = urlopen(req, context=con) File "/home/mike/anaconda3/lib/python3.6/urllib/request.py", line 223, in urlopen return opener.open(url, data, timeout) File "/home/mike/anaconda3/lib/python3.6/urllib/request.py", line 532, in open response = meth(req, response) File "/home/mike/anaconda3/lib/python3.6/urllib/request.py", line 642, in http_response 'http', request, response, code, msg, hdrs) File "/home/mike/anaconda3/lib/python3.6/urllib/request.py", line 570, in error return self._call_chain(*args) File "/home/mike/anaconda3/lib/python3.6/urllib/request.py", line 504, in _call_chain result = func(*args) File "/home/mike/anaconda3/lib/python3.6/urllib/request.py", line 650, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 500: Internal Server Error

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/cli/main.py", line 23, in start cli(obj={}) File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 722, in call return self.main(*args, **kwargs) File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke return callback(*args, **kwargs) File "/home/mike/anaconda3/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func return f(get_current_context(), *args, **kwargs) File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/cli/main.py", line 201, in up dask=dask, notebook=notebook, nprocs=nprocs, source=source) File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke return callback(*args, **kwargs) File "/home/mike/anaconda3/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func return f(get_current_context(), *args, **kwargs) File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/cli/main.py", line 347, in provision ctx.invoke(dask_install, filepath=filepath, nprocs=nprocs, source=source) File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke return callback(*args, **kwargs) File "/home/mike/anaconda3/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func return f(get_current_context(), args, **kwargs) File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/cli/daskd.py", line 80, in dask_install output = cluster.salt_call("node-[1-9]", "state.sls", ["dask.distributed.worker"]) File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/cluster.py", line 65, in salt_call return self.pepper.local(target, module, args) File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/libpepper.py", line 225, in local return self.low([low], path='/') File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/libpepper.py", line 200, in low return self.req(path, lowstate) File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/libpepper.py", line 142, in req raise PepperException('Server error.') dask_ec2.libpepper.PepperException: Server error.

Jan 18 '18 10:01 lionfish0

This might be a duplicate of https://github.com/dask/dask-ec2/issues/30

Jan 18 '18 11:01 lionfish0

dask-ec2 dask-ec2 copied to clipboard

dask_ec2.libpepper.PepperException: Server error.

dask-ec2
dask-ec2 copied to clipboard