flask-testing
flask-testing copied to clipboard
Published
20 hours ago •
jarus
jarus
a lot of tests fail since werkzeug 2.1
tested on current master
venv $ pytest
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.9.12, pytest-7.1.2, pluggy-1.0.0
rootdir: /tmp/flask-testing
collected 41 items
tests/test_twill.py FF [ 4%]
tests/test_utils.py .................F....F......FF..FF.F.. [100%]
========================================================================================== FAILURES ==========================================================================================
_______________________________________________________________________________ TestTwill.test_make_twill_url ________________________________________________________________________________
self = <tests.test_twill.TestTwill testMethod=test_make_twill_url>
def test_make_twill_url(self):
> with Twill(self.app) as t:
tests/test_twill.py:23:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <flask_testing.Error object at 0x7f5fc5718b80>, args = (<Flask 'tests.flask_app'>,), kwargs = {}, msg = "'twill' package is required for Error"
def __init__(self, *args, **kwargs):
msg = "'twill' package is required for %s" % (
self.__class__.__name__)
> raise ImportError(msg)
E ImportError: 'twill' package is required for Error
flask_testing/__init__.py:26: ImportError
_________________________________________________________________________________ TestTwill.test_twill_setup _________________________________________________________________________________
self = <tests.test_twill.TestTwill testMethod=test_twill_setup>
def test_twill_setup(self):
> twill = Twill(self.app)
tests/test_twill.py:16:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <flask_testing.Error object at 0x7f5fc56913a0>, args = (<Flask 'tests.flask_app'>,), kwargs = {}, msg = "'twill' package is required for Error"
def __init__(self, *args, **kwargs):
msg = "'twill' package is required for %s" % (
self.__class__.__name__)
> raise ImportError(msg)
E ImportError: 'twill' package is required for Error
flask_testing/__init__.py:26: ImportError
___________________________________________________________________________ TestClientUtils.test_assert_redirects ____________________________________________________________________________
self = <tests.test_utils.TestClientUtils testMethod=test_assert_redirects>
def test_assert_redirects(self):
response = self.client.get("/redirect/")
> self.assertRedirects(response, "/")
tests/test_utils.py:88:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
flask_testing/utils.py:314: in assertRedirects
self.assertEqual(response.location, expected_location, message)
E AssertionError: '/' != 'http://localhost/'
E - /
E + http://localhost/
__________________________________________________________________ TestClientUtils.test_assert_redirects_valid_status_codes __________________________________________________________________
self = <tests.test_utils.TestClientUtils testMethod=test_assert_redirects_valid_status_codes>
def test_assert_redirects_valid_status_codes(self):
valid_redirect_status_codes = (301, 302, 303, 305, 307)
for status_code in valid_redirect_status_codes:
response = self.client.get("/redirect/?code=" + str(status_code))
> self.assertRedirects(response, "/")
tests/test_utils.py:113:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
flask_testing/utils.py:314: in assertRedirects
self.assertEqual(response.location, expected_location, message)
E AssertionError: '/' != 'http://localhost/'
E - /
E + http://localhost/
__________________________________________________________________________ BaseTestLiveServer.test_server_listening __________________________________________________________________________
self = <tests.test_utils.BaseTestLiveServer testMethod=test_server_listening>, result = <TestCaseFunction test_server_listening>
def __call__(self, result=None):
"""
Does the required setup, doing it here means you don't have to
call super.setUp in subclasses.
"""
# Get the app
> self.app = self.create_app()
flask_testing/utils.py:441:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <tests.test_utils.BaseTestLiveServer testMethod=test_server_listening>
def create_app(self):
"""
Create your Flask app here, with any
configuration you need.
"""
> raise NotImplementedError
E NotImplementedError
flask_testing/utils.py:432: NotImplementedError
_____________________________________________________________________ BaseTestLiveServer.test_server_process_is_spawned ______________________________________________________________________
self = <tests.test_utils.BaseTestLiveServer testMethod=test_server_process_is_spawned>, result = <TestCaseFunction test_server_process_is_spawned>
def __call__(self, result=None):
"""
Does the required setup, doing it here means you don't have to
call super.setUp in subclasses.
"""
# Get the app
> self.app = self.create_app()
flask_testing/utils.py:441:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <tests.test_utils.BaseTestLiveServer testMethod=test_server_process_is_spawned>
def create_app(self):
"""
Create your Flask app here, with any
configuration you need.
"""
> raise NotImplementedError
E NotImplementedError
flask_testing/utils.py:432: NotImplementedError
______________________________________________________________________ TestLiveServerOSPicksPort.test_server_listening _______________________________________________________________________
self = <tests.test_utils.TestLiveServerOSPicksPort testMethod=test_server_listening>, result = <TestCaseFunction test_server_listening>
def __call__(self, result=None):
"""
Does the required setup, doing it here means you don't have to
call super.setUp in subclasses.
"""
# Get the app
self.app = self.create_app()
self._configured_port = self.app.config.get('LIVESERVER_PORT', 5000)
self._port_value = multiprocessing.Value('i', self._configured_port)
# We need to create a context in order for extensions to catch up
self._ctx = self.app.test_request_context()
self._ctx.push()
try:
> self._spawn_live_server()
flask_testing/utils.py:451:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <tests.test_utils.TestLiveServerOSPicksPort testMethod=test_server_listening>
def _spawn_live_server(self):
self._process = None
port_value = self._port_value
def worker(app, port):
# Based on solution: http://stackoverflow.com/a/27598916
# Monkey-patch the server_bind so we can determine the port bound by Flask.
# This handles the case where the port specified is `0`, which means that
# the OS chooses the port. This is the only known way (currently) of getting
# the port out of Flask once we call `run`.
original_socket_bind = socketserver.TCPServer.server_bind
def socket_bind_wrapper(self):
ret = original_socket_bind(self)
# Get the port and save it into the port_value, so the parent process
# can read it.
(_, port) = self.socket.getsockname()
port_value.value = port
socketserver.TCPServer.server_bind = original_socket_bind
return ret
socketserver.TCPServer.server_bind = socket_bind_wrapper
app.run(port=port, use_reloader=False)
self._process = multiprocessing.Process(
target=worker, args=(self.app, self._configured_port)
)
self._process.start()
# We must wait for the server to start listening, but give up
# after a specified maximum timeout
timeout = self.app.config.get('LIVESERVER_TIMEOUT', 5)
start_time = time.time()
while True:
elapsed_time = (time.time() - start_time)
if elapsed_time > timeout:
> raise RuntimeError(
"Failed to start the server after %d seconds. " % timeout
)
E RuntimeError: Failed to start the server after 5 seconds.
flask_testing/utils.py:501: RuntimeError
------------------------------------------------------------------------------------ Captured stdout call ------------------------------------------------------------------------------------
* Serving Flask app 'tests.flask_app' (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
__________________________________________________________________ TestLiveServerOSPicksPort.test_server_process_is_spawned __________________________________________________________________
self = <tests.test_utils.TestLiveServerOSPicksPort testMethod=test_server_process_is_spawned>, result = <TestCaseFunction test_server_process_is_spawned>
def __call__(self, result=None):
"""
Does the required setup, doing it here means you don't have to
call super.setUp in subclasses.
"""
# Get the app
self.app = self.create_app()
self._configured_port = self.app.config.get('LIVESERVER_PORT', 5000)
self._port_value = multiprocessing.Value('i', self._configured_port)
# We need to create a context in order for extensions to catch up
self._ctx = self.app.test_request_context()
self._ctx.push()
try:
> self._spawn_live_server()
flask_testing/utils.py:451:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <tests.test_utils.TestLiveServerOSPicksPort testMethod=test_server_process_is_spawned>
def _spawn_live_server(self):
self._process = None
port_value = self._port_value
def worker(app, port):
# Based on solution: http://stackoverflow.com/a/27598916
# Monkey-patch the server_bind so we can determine the port bound by Flask.
# This handles the case where the port specified is `0`, which means that
# the OS chooses the port. This is the only known way (currently) of getting
# the port out of Flask once we call `run`.
original_socket_bind = socketserver.TCPServer.server_bind
def socket_bind_wrapper(self):
ret = original_socket_bind(self)
# Get the port and save it into the port_value, so the parent process
# can read it.
(_, port) = self.socket.getsockname()
port_value.value = port
socketserver.TCPServer.server_bind = original_socket_bind
return ret
socketserver.TCPServer.server_bind = socket_bind_wrapper
app.run(port=port, use_reloader=False)
self._process = multiprocessing.Process(
target=worker, args=(self.app, self._configured_port)
)
self._process.start()
# We must wait for the server to start listening, but give up
# after a specified maximum timeout
timeout = self.app.config.get('LIVESERVER_TIMEOUT', 5)
start_time = time.time()
while True:
elapsed_time = (time.time() - start_time)
if elapsed_time > timeout:
> raise RuntimeError(
"Failed to start the server after %d seconds. " % timeout
)
E RuntimeError: Failed to start the server after 5 seconds.
flask_testing/utils.py:501: RuntimeError
------------------------------------------------------------------------------------ Captured stdout call ------------------------------------------------------------------------------------
* Serving Flask app 'tests.flask_app' (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
______________________________________________________________ TestNotRenderTemplates.test_assert_template_rendered_signal_sent ______________________________________________________________
self = <tests.test_utils.TestNotRenderTemplates testMethod=test_assert_template_rendered_signal_sent>
def test_assert_template_rendered_signal_sent(self):
self.client.get("/template/")
> self.assert_template_used('index.html')
tests/test_utils.py:259:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
flask_testing/utils.py:244: in assertTemplateUsed
_check_for_signals_support()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
def _check_for_signals_support():
if not _is_signals:
> raise RuntimeError(
"Your version of Flask doesn't support signals. "
"This requires Flask 0.6+ with the blinker module installed."
)
E RuntimeError: Your version of Flask doesn't support signals. This requires Flask 0.6+ with the blinker module installed.
flask_testing/utils.py:112: RuntimeError
====================================================================================== warnings summary ======================================================================================
tests/test_twill.py:27
/tmp/flask-testing/tests/test_twill.py:27: PytestCollectionWarning: cannot collect test class 'TestTwillDeprecated' because it has a __init__ constructor (from: tests/test_twill.py)
class TestTwillDeprecated(TwillTestCase):
../../nix/store/hym1n0ygqp9wcm7pxn4sfrql3fg7xa09-python3-3.9.12/lib/python3.9/unittest/result.py:24
/nix/store/hym1n0ygqp9wcm7pxn4sfrql3fg7xa09-python3-3.9.12/lib/python3.9/unittest/result.py:24: PytestCollectionWarning: cannot collect test class 'TestResult' because it has a __init__ constructor (from: tests/test_utils.py)
class TestResult(object):
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================================== short test summary info ===================================================================================
FAILED tests/test_twill.py::TestTwill::test_make_twill_url - ImportError: 'twill' package is required for Error
FAILED tests/test_twill.py::TestTwill::test_twill_setup - ImportError: 'twill' package is required for Error
FAILED tests/test_utils.py::TestClientUtils::test_assert_redirects - AssertionError: '/' != 'http://localhost/'
FAILED tests/test_utils.py::TestClientUtils::test_assert_redirects_valid_status_codes - AssertionError: '/' != 'http://localhost/'
FAILED tests/test_utils.py::BaseTestLiveServer::test_server_listening - NotImplementedError
FAILED tests/test_utils.py::BaseTestLiveServer::test_server_process_is_spawned - NotImplementedError
FAILED tests/test_utils.py::TestLiveServerOSPicksPort::test_server_listening - RuntimeError: Failed to start the server after 5 seconds.
FAILED tests/test_utils.py::TestLiveServerOSPicksPort::test_server_process_is_spawned - RuntimeError: Failed to start the server after 5 seconds.
FAILED tests/test_utils.py::TestNotRenderTemplates::test_assert_template_rendered_signal_sent - RuntimeError: Your version of Flask doesn't support signals. This requires Flask 0.6+ with ...
========================================================================= 9 failed, 32 passed, 2 warnings in 10.47s ==========================================================================
[1] /tmp/flask-testing master 11s
venv $ pip freeze
attrs==21.4.0
backports.entry-points-selectable @ file:///build/backports.entry_points_selectable-1.1.1/dist/backports.entry_points_selectable-1.1.1-py2.py3-none-any.whl
click==8.1.3
distlib @ file:///build/distlib-0.3.4/dist/distlib-0.3.4-py3-none-any.whl
filelock @ file:///build/filelock-3.6.0/dist/filelock-3.6.0-py3-none-any.whl
Flask==2.1.2
Flask-Testing==0.8.1
importlib-metadata==4.11.3
iniconfig==1.1.1
itsdangerous==2.1.2
Jinja2==3.1.2
MarkupSafe==2.1.1
packaging==21.3
platformdirs @ file:///build/source/dist/platformdirs-2.5.1-py3-none-any.whl
pluggy==1.0.0
py==1.11.0
pyparsing==3.0.8
pytest==7.1.2
six @ file:///build/six-1.16.0/dist/six-1.16.0-py2.py3-none-any.whl
tomli==2.0.1
virtualenv @ file:///build/virtualenv-20.14.0/dist/virtualenv-20.14.0-py2.py3-none-any.whl
Werkzeug==2.1.2
zipp==3.8.0
I have also run into this. It appears that at least in our case, the RuntimeError: Failed to start the server after 5 seconds. is triggered when setting LIVESERVER_PORT to 0. In particular, the following code is never invoked, which makes me suspect the patch it's doing is failing: https://github.com/jarus/flask-testing/blob/5107691011fa891835c01547e73e991c484fa07f/flask_testing/utils.py#L474
I can work around it by setting a specific port, but that isn't very resilient.
