PepperException: Server error

[jochenschreiber]

Can somebody help me out to understand why we sometimes got the following exception:

File "/usr/local/lib/python2.7/dist-packages/pepper/libpepper.py", line 297, in low
    return self.req(path, lowstate)
  File "/usr/local/lib/python2.7/dist-packages/pepper/libpepper.py", line 241, in req
    raise PepperException('Server error.')
PepperException: Server error.

My assumption is that this comes from any long running jobs but we have already increased the timeouts inside our salt-master config:

rest_cherrypy:
  port: 8000
  disable_ssl: True
  debug: True
  rest_timeout: 7200
  timeout: 7200
  expire_responses: False

What can be used to debug this? As i mentioned that only happens sometimes. If you may run the same salt call again it works

[gtmanfred]

What command are you running? and how long is the job running?

[jochenschreiber]

As far as i can see it happens when we run our highstate. The highstate takes ~15 minutes.

[icksa]

I've also started running into this "Server error." message. I believe it is due to a timeout on the server. The api log shows this when the error occurs:

2019-03-06 10:32:27,177 [cherrypy.error   ][ERROR   ][1772] [06/Mar/2019:10:32:27]  Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/cherrypy/_cpwsgi.py", line 169, in trap
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/cherrypy/_cpwsgi.py", line 96, in __call__
    return self.nextapp(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/cherrypy/_cpwsgi.py", line 379, in tail
    return self.response_class(environ, start_response, self.cpapp)
  File "/usr/local/lib/python2.7/dist-packages/cherrypy/_cpwsgi.py", line 222, in __init__
    self.run()
  File "/usr/local/lib/python2.7/dist-packages/cherrypy/_cpwsgi.py", line 320, in run
    request.run(meth, path, qs, rproto, headers, rfile)
  File "/usr/local/lib/python2.7/dist-packages/cherrypy/_cprequest.py", line 603, in run
    raise cherrypy.TimeoutError()
TimeoutError

From what I can tell, the recommendation from the salt people is to pass an explicit timeout parameter when making a salt call. A few other PRs have pointed out that passing the timeout parameter to pepper does not add it to the arguments sent to the API server.

[gtmanfred]

Are you seeing this on 2019.2?

because i am seeing some errors in the test suite caused by 2019.2

[icksa]

I'm actually running pretty old salt and cherry py... (2017.7 and 3.2.3 respectively)

[prashanthbgoud]

I see the same issue on salt 2018.3.4 (Oxygen).