nose
nose copied to clipboard
nose-devs
Multiprocess KeyErrors that result in broken pipes
We run nose with python selenium and occasionally hit a nose issue with KeyErrors that then turn into broken pipes and we can't recover from.
Python 2.6.6 nosetests --goober --processes=1 --process-timeout=300 --process-restartworker --with-xunitmp Debian 6.0.10
Is this a nose bug? Memory corruption? Something we should be doing to narrow this down?
/usr/local/lib/python2.6/dist-packages/pysel/webdriver.py:146:
.........................................................................................................F......S....................................................................................................Traceback (most recent call last):
File "/usr/local/bin/nosetests", line 9, in <module>
load_entry_point('nose==1.3.7', 'console_scripts', 'nosetests')()
File "/usr/local/lib/python2.6/dist-packages/nose/core.py", line 121, in __init__
**extra_args)
File "/usr/lib/python2.6/unittest.py", line 817, in __init__
self.runTests()
File "/usr/local/lib/python2.6/dist-packages/nose/core.py", line 207, in runTests
result = self.testRunner.run(self.test)
File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 474, in run
if w.is_alive() and len(w.currentaddr.value) > 0:
File "/usr/lib/python2.6/multiprocessing/managers.py", line 1005, in get
return self._callmethod('get')
File "/usr/lib/python2.6/multiprocessing/managers.py", line 740, in _callmethod
raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:
---------------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.6/multiprocessing/managers.py", line 216, in serve_client
obj, exposed, gettypeid = id_to_obj[ident]
KeyError: '1a033d0'
---------------------------------------------------------------------------
Process Process-252:
Traceback (most recent call last):
File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
self.run()
File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 652, in runner
keyboardCaught, shouldStop, loaderClass, resultClass, config)
File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 745, in __runner
currentaddr.value = bytes_('')
File "/usr/lib/python2.6/multiprocessing/managers.py", line 1007, in set
return self._callmethod('set', (value,))
File "/usr/lib/python2.6/multiprocessing/managers.py", line 740, in _callmethod
raise convert_to_error(kind, result)
RemoteError:
---------------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.6/multiprocessing/managers.py", line 216, in serve_client
obj, exposed, gettypeid = id_to_obj[ident]
KeyError: '1a001d0'
---------------------------------------------------------------------------
Process Process-251:
Traceback (most recent call last):
File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
Process Process-253:
Traceback (most recent call last):
File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
self.run()
File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 652, in runner
self.run()
File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 652, in runner
keyboardCaught, shouldStop, loaderClass, resultClass, config)
File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 745, in __runner
currentaddr.value = bytes_('')
File "/usr/lib/python2.6/multiprocessing/managers.py", line 1007, in set
keyboardCaught, shouldStop, loaderClass, resultClass, config)
File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 745, in __runner
currentaddr.value = bytes_('')
File "/usr/lib/python2.6/multiprocessing/managers.py", line 1007, in set
return self._callmethod('set', (value,))
File "/usr/lib/python2.6/multiprocessing/managers.py", line 740, in _callmethod
raise convert_to_error(kind, result)
RemoteError:
---------------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.6/multiprocessing/managers.py", line 216, in serve_client
obj, exposed, gettypeid = id_to_obj[ident]
KeyError: '1a70050'
---------------------------------------------------------------------------
return self._callmethod('set', (value,))
File "/usr/lib/python2.6/multiprocessing/managers.py", line 740, in _callmethod
raise convert_to_error(kind, result)
RemoteError:
---------------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.6/multiprocessing/managers.py", line 216, in serve_client
obj, exposed, gettypeid = id_to_obj[ident]
KeyError: '1a6aa90'
---------------------------------------------------------------------------
Process Process-254:
Traceback (most recent call last):
File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
self.run()
File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 652, in runner
keyboardCaught, shouldStop, loaderClass, resultClass, config)
File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 745, in __runner
currentaddr.value = bytes_('')
File "/usr/lib/python2.6/multiprocessing/managers.py", line 1007, in set
return self._callmethod('set', (value,))
File "/usr/lib/python2.6/multiprocessing/managers.py", line 740, in _callmethod
raise convert_to_error(kind, result)
RemoteError:
---------------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.6/multiprocessing/managers.py", line 216, in serve_client
obj, exposed, gettypeid = id_to_obj[ident]
KeyError: '1a033d0'
---------------------------------------------------------------------------
Process Process-255:
Traceback (most recent call last):
File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
self.run()
File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 652, in runner
keyboardCaught, shouldStop, loaderClass, resultClass, config)
File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 700, in __runner
for test_addr, arg in iter(get, 'STOP'):
File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 675, in get
return testQueue.get(timeout=config.multiprocess_timeout)
File "<string>", line 2, in get
File "/usr/lib/python2.6/multiprocessing/managers.py", line 722, in _callmethod
self._connect()
File "/usr/lib/python2.6/multiprocessing/managers.py", line 709, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
File "/usr/lib/python2.6/multiprocessing/connection.py", line 143, in Client
c = SocketClient(address)
File "/usr/lib/python2.6/multiprocessing/connection.py", line 263, in SocketClient
s.connect(address)
File "<string>", line 1, in connect
error: [Errno 2] No such file or directory
Process Process-214:
Traceback (most recent call last):
File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
self.run()
File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 652, in runner
keyboardCaught, shouldStop, loaderClass, resultClass, config)
File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 745, in __runner
currentaddr.value = bytes_('')
File "/usr/lib/python2.6/multiprocessing/managers.py", line 1007, in set
return self._callmethod('set', (value,))
File "/usr/lib/python2.6/multiprocessing/managers.py", line 725, in _callmethod
conn.send((self._id, methodname, args, kwds))
IOError: [Errno 32] Broken pipe
Process Process-223:
Traceback (most recent call last):
File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
self.run()
File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 652, in runner
keyboardCaught, shouldStop, loaderClass, resultClass, config)
File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 745, in __runner
currentaddr.value = bytes_('')
File "/usr/lib/python2.6/multiprocessing/managers.py", line 1007, in set
return self._callmethod('set', (value,))
File "/usr/lib/python2.6/multiprocessing/managers.py", line 725, in _callmethod
conn.send((self._id, methodname, args, kwds))
IOError: [Errno 32] Broken pipe
Seeing similar thing with Python 2.7.6 and Nose 1.3.7. Have you had any luck diagnosing? FWIW I don't see it on a box with Python 2.7.4 and Nose 1.3.0.
nose.plugins.multiprocess: ERROR: Worker 5 error running test or returning results
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/nose/plugins/multiprocess.py", line 714, in __runner
currentstart.value = time.time()
File "/usr/local/lib/python2.7/multiprocessing/managers.py", line 1041, in set
return self._callmethod('set', (value,))
File "/usr/local/lib/python2.7/multiprocessing/managers.py", line 774, in _callmethod
raise convert_to_error(kind, result)
RemoteError:
---------------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python2.7/multiprocessing/managers.py", line 242, in serve_client
obj, exposed, gettypeid = id_to_obj[ident]
KeyError: '2539190'
---------------------------------------------------------------------------
Failure: RemoteError (
---------------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python2.7/multiprocessing/managers.py", line 242, in serve_client
obj, exposed, gettypeid = id_to_obj[ident]
KeyError: '2539190'
---------------------------------------------------------------------------) ... ERROR
Traceback (most recent call last):
File "/usr/local/bin/nosetests", line 9, in <module>
load_entry_point('nose==1.3.7', 'console_scripts', 'nosetests')()
File "/usr/local/lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/nose/core.py", line 121, in __init__
**extra_args)
File "/usr/local/lib/python2.7/unittest/main.py", line 95, in __init__
self.runTests()
File "/usr/local/lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/nose/core.py", line 207, in runTests
result = self.testRunner.run(self.test)
File "/usr/local/lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/nose/plugins/multiprocess.py", line 474, in run
if w.is_alive() and len(w.currentaddr.value) > 0:
File "/usr/local/lib/python2.7/multiprocessing/managers.py", line 1039, in get
return self._callmethod('get')
File "/usr/local/lib/python2.7/multiprocessing/managers.py", line 774, in _callmethod
raise convert_to_error(kind, result)
No we worked around the problem by changing our tests so the dictionaries weren’t as large and that seemed to help. Would love to know if you find any more information since I’m sure we’ll run into it again.
On August 10, 2016 at 7:17:54 AM, Jonathan Ross ([email protected]) wrote:
Seeing similar thing with Python 2.7.6 and Nose 1.3.7. Have you had any luck diagnosing?
nose.plugins.multiprocess: ERROR: Worker 5 error running test or returning results Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/nose/plugins/multiprocess.py", line 714, in __runner currentstart.value = time.time() File "/usr/local/lib/python2.7/multiprocessing/managers.py", line 1041, in set return self._callmethod('set', (value,)) File "/usr/local/lib/python2.7/multiprocessing/managers.py", line 774, in _callmethod raise convert_to_error(kind, result)
RemoteError:
Traceback (most recent call last): File "/usr/local/lib/python2.7/multiprocessing/managers.py", line 242, in serve_client obj, exposed, gettypeid = id_to_obj[ident]
KeyError: '2539190'
Failure: RemoteError (
Traceback (most recent call last):
File "/usr/local/lib/python2.7/multiprocessing/managers.py", line
242, in serve_client
obj, exposed, gettypeid = id_to_obj[ident]
KeyError: '2539190'
---------------------------------------------------------------------------)
... ERROR
Traceback (most recent call last):
File "/usr/local/bin/nosetests", line 9, in
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/nose-devs/nose/issues/997#issuecomment-238880719, or mute the thread https://github.com/notifications/unsubscribe-auth/AC4PLID8Ano8iii3aAFSD4NmtxvdH5cVks5qed2SgaJpZM4IYMZj .
@mjmj apologies for hijacking the thread but since i'm new and it would be great if you can point me in right direct on how did you make "---goober" flag working? I ran setup.py but I get no such option: --goober
no worries @mjmj , got to from the author dustinkeitel using (pip install -e .)