pytest-bdd
pytest-bdd copied to clipboard
pytest-dev
Async Step Definitions
is there away to use async step definitions with pytest-bdd ?
For a step definition like:
@when('i send cucumbers ')
async def i_send_cucumbers(loop):
pass
I see the warning:
pytest_bdd/scenario.py:137: RuntimeWarning: coroutine 'i_send_cucumbers' was never awaited
will this work for you?
async def send_cucumbers(): pass
@when('i send cucumbers ') def i_send_cucumbers(loop): loop.run_until_complete(send_cucumbers)
@olegpidsadnyi This works, however, in my case all step definitions are async (and can call multiple times into async functions). so this would mean a lot of calls to run_until_complete. As a work-around I have defined a decorator that schedules the function on the event loop.
A generic solution would be to have the ability to easily wrap any step definition with a function. (similar to the before and after hooks).
And then it would be great if support of async functions would be provided out of the box.
Especially, after reading the discussion about implicit and explicit eventloops in the thread here: https://groups.google.com/forum/#!msg/python-tulip/yF9C-rFpiKk/tk5oA3GLHAAJ seems to be tending towards implicit loops for python code that runs on 3.5.3+ and/or 3.6+.
So code that explicitly calls run_until_complete to run async functions will look more and more awkward.
@jeduden this means pytest-bdd has to depend on yet another library asyncio? I don't really see how a gherkin scenario can by async. If the whole point that it has not to break i think you need to extend your testing suite with some kind of support for async functions (decorator is a good idea).
I think semantically you can't really represent async process in step-by-step imperative Gherkin. Therefore it should be explicit. If you have to initiate few async messages - they should stand behind one When step that describes this exercise in a form that humans understand. Gherkin is not a programming language, it is the way to describe steps to humans which can't do async computation in their brains.
What do you think, @bubenkoff ?
well, the async question comes down to the pytest, not specifically to pytest-bdd. And for pytest's dependency injection, it's not realistic that it starts to support async fixture definitions anytime soon. Also for tests, while it sounds cool there seems to be a little win to have async fixtures, simply because fixtures should be fast enough, and then it should not matter much if you optimise dependency graph in a way that you run fixtures in parallel for some parts of the graph. It worth effort though to add the 'async clause' to the documentation mentining the workaround to cut the async point on the fixture definition by waiting for async function to finish
@jeduden could you show your decorator implementation for when step that is awaiting?
FYI: there's a helper apparently to minimize the efforts: https://pypi.python.org/pypi/pytest-asyncio
Support on pytest side is fine. We are using aiohttp test-support and with pytest-asyncio 0.8.0 we can also the helpers from that package.
However, these helpers work on scenario level.
The step_definitions are one level below, and since the calling code here: https://github.com/pytest-dev/pytest-bdd/blob/master/pytest_bdd/scenario.py#L137 is not checking if step_func is async. i.e. it only supports synchronous step functions.
In order to fix this, we currently need to wrap all steps with decorators like this:
def sync(func):
@wraps(func)
def synced_func(*args, **kwargs):
loop = kwargs.get("loop")
if not loop:
raise Exception("Need loop fixture to make function sync")
return loop.run_until_complete(func(*args, **kwargs))
return synced_func
@olegpidsadnyi declaring a function async doesn't mean it is not imperative. Async functions are imperative like synchronous functions. However, with await / yield from you explicitly define points where the execution of other scheduled coroutines is allowed.
In order to schedule the parallel execution of multiple asynchronous functions you would use helpers like asyncio.gather ( see https://docs.python.org/3/library/asyncio-task.html#example-parallel-execution-of-tasks )
The reason why we need to declare our step_definitions async is because we are using the aiohttp client inside to perform networking calls as part of our step definitions. Furthermore, the rest of the code base is completely async, hence we think it is only natural that also step_defintions are written with async.
As i see from the pytest-asyncio code, it just awaits for every 'async' fixture, so there's no real parallelism possible between the fixtures And how do you mean pytest upports async fixtures in a 'proper' way? Can't really see that
But i do see your point about the need of the wrapper everywhere - it sucks. Looks like the only way to avoid that is to depend on pytest-asyncio and use it's helpers directly in the pytest-bdd
@bubenkoff even that is not possible since there is no hook that allows me to insert a custom decorator on the functions.
even if pytest-bdd doesn't support async step definitions out of the box. A hook for wrapping a step definitions would help to reduce duplication.
are you up for making a PR which will add pytest-asyncio as a dependency and automatically use it to resolve async step definitions, if they are async?
@bubenkoff In general, I am happy to make a PR. One question regarding the pytest-asyncio dependency. You mention it because we need to have a library to provide the loop fixture, right ?
@jeduden yes, also to keep as much core async stuff as possible in a single plugin (pytest-asyncio)
@bubenkoff i did a quick check on the current test-suite; how i can best write some this for this new type of possible step functions. do you have a pointer for me where i best can add tests ? Or should i create a complete new file ?
You can put a new file here tests/steps/test_async.py copying the approach of tests/steps/test_unicode.py and replacing definitions with async ones:
@given
async def ...
@when
async def ...
@then
async def ...
For what its worth, I took the approach with #221 and then added that hook implementation as a separate pytest plugin that I can install as necessary.
@vodik
added that hook implementation as a separate pytest plugin
Is your pytest plugin available anywhere, just so I can try it out? Do you plan on maintaining it? My gut feeling is that integrating the functionality into pytest-bdd would be the path of least maintenance burden, but that is up to the maintainers I suppose.
First off, thank you so much for this awesome library! I was hoping I can contribute my experience with asyncio.
@olegpidsadnyi sad here:
this means pytest-bdd has to depend on yet another library asyncio?
Depends on how you look at it. asyncio is part of the standard library since 3.4. So users wouldn't need to install anything special. This feature is not needed by users of older versions of python since they don't use the async/await syntax to begin with, so they have no need for it.
I think semantically you can't really represent async process in step-by-step imperative Gherkin.
Agreed, but the cool thing is that despite the name, async/await doesn't necessarily mean that the operations happen in arbitrary order. await just means that async operations suspend the current execution thread (not system thread, mind you) and execution is resumed once they are completed, in the same order as specified in the function. This way, the semantics of the operations are unchanged happen in a step-by-step imperative mode.
Please let me know if you would like to have a conversation about this topic. I'm always happy to talk about async/await :)
Behave 1.2.6 added some decorators to Testing asyncio Frameworks .
That may be inspirational to implement something similar for pytest-bdd.
Is there a guide somewhere to using pytest-bdd with pytest-asyncio ? I'm not sure how to make them play nicely together and actually execute my bdd tests.
Just shooting this question as I am stuck with the same issue for handling async step definitions as pytest-asyncio cannot be used here.
Is there some update or any way to handle the same as of now? Hope you will please help if some new changes/ways to handle are present.
@bubenkoff
Most of web development nowadays is moving to async frameworks (for instance fastapi). Testing this frameworks typically involves running async app and also use an async testclient.
Support for async pytest-bdd step definitions would definitely help in further adoption in these kind of projects.
I forked the project and applied PR. I was able to take advantage of asynchronous tests. But as time passes, I see that this is unnecessary. If you write integration tests, then nothing prevents you from using requests as a client. I will support the voiced idea that adding asynchrony is redundant and solves other problem.
@DjaPy but what about if your integration tests are part of your whole asyncio project with lots of async tests? Isn't it's better to just run pytest on the whole project and view all test results at once?
I thought that's one of the key advantages of pytest-bdd. Otherwise, if you write your bdd tests fully isolated from other project test structure (async fixtures for example), what's the benefit then of using pytest-bdd instead of let's say behave?
Currently, I actually do as you suggested, by separating bdd tests (using requests in them) and other tests. However again, for local development, for CI/CD I always need to run two commands instead of one. I mean it's not probably a "must-have" feature, but definitely "nice-to-have" for pytest-bdd.
BTW want to give my warm thanks for an awesome library :rocket:
