channels
channels copied to clipboard
Segfault when running ChannelsLiveServerTestCase
I am using channels 2.3.1 with daphne 2.4.0 - which I can confirm fixed https://github.com/django/channels/issues/962. Unfortunately, while it doesn't hang anymore, it segfaults when the server connects to a PostgreSQL server with psycopg2 (built on OSX, not using the provided binaries).
It only happens specifically on OSX and when running a ChannelsLiveServerTestCase - not on debian or when using runserver.
I'm not able to provide logs as of now, but if you need it I can provide them tomorrow, with a minimal project to try and reproduce this issue.
Minimal reproduce, and actual error would be handy. Thanks.
Okay, with a minimal project the traceback is way more readable. It seems you can access static files just fine, but it crashes when accessing the database. Since the daphne server crashes, selenium fails to load the page and the test fails.
Project : https://github.com/kiwec/channels-segfault
Crash log :
============================= test session starts ==============================
platform darwin -- Python 3.7.3, pytest-5.3.1, py-1.8.0, pluggy-0.13.1
rootdir: /Users/clementwolf/Documents/channels-segfault
collected 2 items
channelsegfault/test.py .Fatal Python error: Segmentation fault
Current thread 0x00007000015db000 (most recent call first):
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/psycopg2/__init__.py", line 126 in connect
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/django/db/backends/postgresql/base.py", line 178 in get_new_connection
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/django/db/backends/base/base.py", line 195 in connect
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/django/db/backends/base/base.py", line 217 in ensure_connection
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/django/db/backends/base/base.py", line 233 in _cursor
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/django/db/backends/base/base.py", line 256 in cursor
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1366 in execute_sql
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/django/db/models/query.py", line 1186 in _insert
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/django/db/models/manager.py", line 82 in manager_method
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/django/db/models/base.py", line 908 in _do_insert
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/django/db/models/base.py", line 870 in _save_table
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/django/db/models/base.py", line 779 in save_base
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/django/db/models/base.py", line 741 in save
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/django/db/models/query.py", line 422 in create
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/django/db/models/manager.py", line 82 in manager_method
File "/Users/clementwolf/Documents/channels-segfault/channelsegfault/main.py", line 21 in crash_my_shit_up_view
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/django/core/handlers/base.py", line 113 in _get_response
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34 in inner
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/django/core/handlers/base.py", line 75 in get_response
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/channels/http.py", line 244 in handle
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/asgiref/sync.py", line 277 in thread_handler
File "/Users/clementwolf/.pyenv/versions/3.7.3/lib/python3.7/concurrent/futures/thread.py", line 57 in run
File "/Users/clementwolf/.pyenv/versions/3.7.3/lib/python3.7/concurrent/futures/thread.py", line 80 in _worker
File "/Users/clementwolf/.pyenv/versions/3.7.3/lib/python3.7/threading.py", line 865 in run
File "/Users/clementwolf/.pyenv/versions/3.7.3/lib/python3.7/threading.py", line 917 in _bootstrap_inner
File "/Users/clementwolf/.pyenv/versions/3.7.3/lib/python3.7/threading.py", line 885 in _bootstrap
Thread 0x0000000109aed5c0 (most recent call first):
File "/Users/clementwolf/.pyenv/versions/3.7.3/lib/python3.7/selectors.py", line 558 in select
File "/Users/clementwolf/.pyenv/versions/3.7.3/lib/python3.7/asyncio/base_events.py", line 1739 in _run_once
File "/Users/clementwolf/.pyenv/versions/3.7.3/lib/python3.7/asyncio/base_events.py", line 539 in run_forever
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/twisted/internet/asyncioreactor.py", line 267 in run
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/daphne/server.py", line 140 in run
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/daphne/testing.py", line 144 in run
File "/Users/clementwolf/.pyenv/versions/3.7.3/lib/python3.7/multiprocessing/process.py", line 297 in _bootstrap
File "/Users/clementwolf/.pyenv/versions/3.7.3/lib/python3.7/multiprocessing/popen_fork.py", line 74 in _launch
File "/Users/clementwolf/.pyenv/versions/3.7.3/lib/python3.7/multiprocessing/popen_fork.py", line 20 in __init__
File "/Users/clementwolf/.pyenv/versions/3.7.3/lib/python3.7/multiprocessing/context.py", line 277 in _Popen
File "/Users/clementwolf/.pyenv/versions/3.7.3/lib/python3.7/multiprocessing/context.py", line 223 in _Popen
File "/Users/clementwolf/.pyenv/versions/3.7.3/lib/python3.7/multiprocessing/process.py", line 112 in start
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/channels/testing/live.py", line 52 in _pre_setup
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/django/test/testcases.py", line 267 in __call__
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/_pytest/unittest.py", line 207 in runtest
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/_pytest/runner.py", line 131 in pytest_runtest_call
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/_pytest/runner.py", line 207 in <lambda>
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/_pytest/runner.py", line 234 in from_call
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/_pytest/runner.py", line 207 in call_runtest_hook
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/_pytest/runner.py", line 182 in call_and_report
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/_pytest/runner.py", line 96 in runtestprotocol
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/_pytest/runner.py", line 81 in pytest_runtest_protocol
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/_pytest/main.py", line 270 in pytest_runtestloop
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/_pytest/main.py", line 246 in _main
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/_pytest/main.py", line 196 in wrap_session
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/_pytest/main.py", line 239 in pytest_cmdline_main
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/_pytest/config/__init__.py", line 92 in main
File "/Users/clementwolf/Documents/channels-segfault/channelsegfault/main.py", line 17 in run_tests
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/django/core/management/commands/test.py", line 53 in handle
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/django/core/management/base.py", line 364 in execute
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/django/core/management/base.py", line 323 in run_from_argv
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/django/core/management/commands/test.py", line 23 in run_from_argv
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/django/core/management/__init__.py", line 375 in execute
File "/Users/clementwolf/Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/django/core/management/__init__.py", line 381 in execute_from_command_line
File "manage.py", line 17 in main
File "manage.py", line 21 in <module>
F [100%]
=================================== FAILURES ===================================
________________________ AmazingTestCase.test_the_stuff ________________________
self = <channelsegfault.test.AmazingTestCase testMethod=test_the_stuff>
def test_the_stuff(self):
DummyModel.objects.create(content="no crash here")
self.browser.implicitly_wait(10)
> self.browser.get(self.live_server_url)
channelsegfault/test.py:33:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py:333: in get
self.execute(Command.GET, {'url': url})
../../Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py:321: in execute
self.error_handler.check_response(response)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <selenium.webdriver.remote.errorhandler.ErrorHandler object at 0x1097fcc50>
response = {'status': 500, 'value': '{"value":{"error":"unknown error","message":"Reached error page: about:neterror?e=dnsNotFoun...ate@chrome://marionette/content/listener.js:297:13\\nhandleEvent@chrome://marionette/content/listener.js:263:14\\n"}}'}
def check_response(self, response):
"""
Checks that a JSON response from the WebDriver does not have an error.
:Args:
- response - The JSON response from the WebDriver server as a dictionary
object.
:Raises: If the response contains an error message.
"""
status = response.get('status', None)
if status is None or status == ErrorCode.SUCCESS:
return
value = None
message = response.get("message", "")
screen = response.get("screen", "")
stacktrace = None
if isinstance(status, int):
value_json = response.get('value', None)
if value_json and isinstance(value_json, basestring):
import json
try:
value = json.loads(value_json)
if len(value.keys()) == 1:
value = value['value']
status = value.get('error', None)
if status is None:
status = value["status"]
message = value["value"]
if not isinstance(message, basestring):
value = message
message = message.get('message')
else:
message = value.get('message', None)
except ValueError:
pass
exception_class = ErrorInResponseException
if status in ErrorCode.NO_SUCH_ELEMENT:
exception_class = NoSuchElementException
elif status in ErrorCode.NO_SUCH_FRAME:
exception_class = NoSuchFrameException
elif status in ErrorCode.NO_SUCH_WINDOW:
exception_class = NoSuchWindowException
elif status in ErrorCode.STALE_ELEMENT_REFERENCE:
exception_class = StaleElementReferenceException
elif status in ErrorCode.ELEMENT_NOT_VISIBLE:
exception_class = ElementNotVisibleException
elif status in ErrorCode.INVALID_ELEMENT_STATE:
exception_class = InvalidElementStateException
elif status in ErrorCode.INVALID_SELECTOR \
or status in ErrorCode.INVALID_XPATH_SELECTOR \
or status in ErrorCode.INVALID_XPATH_SELECTOR_RETURN_TYPER:
exception_class = InvalidSelectorException
elif status in ErrorCode.ELEMENT_IS_NOT_SELECTABLE:
exception_class = ElementNotSelectableException
elif status in ErrorCode.ELEMENT_NOT_INTERACTABLE:
exception_class = ElementNotInteractableException
elif status in ErrorCode.INVALID_COOKIE_DOMAIN:
exception_class = InvalidCookieDomainException
elif status in ErrorCode.UNABLE_TO_SET_COOKIE:
exception_class = UnableToSetCookieException
elif status in ErrorCode.TIMEOUT:
exception_class = TimeoutException
elif status in ErrorCode.SCRIPT_TIMEOUT:
exception_class = TimeoutException
elif status in ErrorCode.UNKNOWN_ERROR:
exception_class = WebDriverException
elif status in ErrorCode.UNEXPECTED_ALERT_OPEN:
exception_class = UnexpectedAlertPresentException
elif status in ErrorCode.NO_ALERT_OPEN:
exception_class = NoAlertPresentException
elif status in ErrorCode.IME_NOT_AVAILABLE:
exception_class = ImeNotAvailableException
elif status in ErrorCode.IME_ENGINE_ACTIVATION_FAILED:
exception_class = ImeActivationFailedException
elif status in ErrorCode.MOVE_TARGET_OUT_OF_BOUNDS:
exception_class = MoveTargetOutOfBoundsException
elif status in ErrorCode.JAVASCRIPT_ERROR:
exception_class = JavascriptException
elif status in ErrorCode.SESSION_NOT_CREATED:
exception_class = SessionNotCreatedException
elif status in ErrorCode.INVALID_ARGUMENT:
exception_class = InvalidArgumentException
elif status in ErrorCode.NO_SUCH_COOKIE:
exception_class = NoSuchCookieException
elif status in ErrorCode.UNABLE_TO_CAPTURE_SCREEN:
exception_class = ScreenshotException
elif status in ErrorCode.ELEMENT_CLICK_INTERCEPTED:
exception_class = ElementClickInterceptedException
elif status in ErrorCode.INSECURE_CERTIFICATE:
exception_class = InsecureCertificateException
elif status in ErrorCode.INVALID_COORDINATES:
exception_class = InvalidCoordinatesException
elif status in ErrorCode.INVALID_SESSION_ID:
exception_class = InvalidSessionIdException
elif status in ErrorCode.UNKNOWN_METHOD:
exception_class = UnknownMethodException
else:
exception_class = WebDriverException
if value == '' or value is None:
value = response['value']
if isinstance(value, basestring):
if exception_class == ErrorInResponseException:
raise exception_class(response, value)
raise exception_class(value)
if message == "" and 'message' in value:
message = value['message']
screen = None
if 'screen' in value:
screen = value['screen']
stacktrace = None
if 'stackTrace' in value and value['stackTrace']:
stacktrace = []
try:
for frame in value['stackTrace']:
line = self._value_or_default(frame, 'lineNumber', '')
file = self._value_or_default(frame, 'fileName', '<anonymous>')
if line:
file = "%s:%s" % (file, line)
meth = self._value_or_default(frame, 'methodName', '<anonymous>')
if 'className' in frame:
meth = "%s.%s" % (frame['className'], meth)
msg = " at %s (%s)"
msg = msg % (meth, file)
stacktrace.append(msg)
except TypeError:
pass
if exception_class == ErrorInResponseException:
raise exception_class(response, message)
elif exception_class == UnexpectedAlertPresentException:
alert_text = None
if 'data' in value:
alert_text = value['data'].get('text')
elif 'alert' in value:
alert_text = value['alert'].get('text')
raise exception_class(message, screen, stacktrace, alert_text)
> raise exception_class(message, screen, stacktrace)
E selenium.common.exceptions.WebDriverException: Message: Reached error page: about:neterror?e=dnsNotFound&u=http%3A//localhost%3A49239/&c=UTF-8&f=regular&d=Impossible%20de%20se%20connecter%20au%20serveur%20%C3%A0%20l%E2%80%99adresse%20localhost.
../../Library/Caches/pypoetry/virtualenvs/channels-segfault-py3.7/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py:242: WebDriverException
========================= 1 failed, 1 passed in 8.94s ==========================
~Changed pytest to unittest and just worked fine. I think this issue has something to do with Django + multiprocessing + pytest combination, not Channels or Daphne itself.~ Sorry, my mistake. The problem still exists with unittest.
After several experiments, it turned out that this is a problem of psycopg2 + multiprocessing. The following script also crashes with SIGSEGV in MacOS.
import psycopg2
import multiprocessing
def psycopg2_connect():
psycopg2.connect(host='localhost', port='5432', database='postgres')
print('Done') # This is not printed because the process crashes
c = psycopg2.connect(host='localhost', port='5432', database='postgres')
c.close()
p = multiprocessing.Process(target=psycopg2_connect)
p.start()
p.join()
The version of psycopg2 is 2.8.4.
This seems to be related to psycopg/psycopg2#691. I have no idea what directly caused the problem nor how to fix it, but i believe that there is nothing Channels or Daphne can do. A workaround can be using a different database or just stick to Linux. FYI @kiwec @carltongibson
@joonhyungshin Can you try my pull requests for Daphne and Channels and see if that fixes your problem?
Daphne: https://github.com/django/daphne/pull/284 Channels: https://github.com/django/channels/pull/1372
It uses threads instead of separate processes.
If it does fix your problem, then maybe Daphne and Channels can accept it to prevent issues like this happening for other libraries, since Python Processes and third party libraries don't always mesh well together.
Closing based on https://github.com/django/channels/issues/1387#issuecomment-565734357
Ref more general issues here, I've merged #1906 to enforce fork
for the moment, which at least lets test cases work on macOS. (We'll need a fuller solution long-term, and for Windows support.)
Note pre-4.0 this requires the development version of daphne
as well. See #1898