Sync wrapper for hypothesis tests uses the event loop provided by pytest-asyncio

[seifertm]

Async test functions marked for execution by Hypothesis are currently wrapped in a synchronous function during the test discovery phase of pytest. At this point, the event_loop fixture provided by pytest-asyncio has not been evaluated and the wrapper is forced to create a new loop for each hypothesis test. This may result is problems when using async-related code in fixtures as in #117.

This PR moves the wrapping code to the actual test execution. At this point the event_loop fixture is evaluated and there is no need to create a new loop for the test.

What's missing from this PR is a proper automated test case. Since I have run into this issue using aiohttp, I was using the following code to test it:

import aiohttp
import pytest
from hypothesis import given
from hypothesis.strategies import integers


@pytest.fixture
async def http():
    async with aiohttp.ClientSession() as session:
        yield session


@pytest.mark.asyncio
@given(integers())
async def test_connect(http, n):
    await http.get('http://127.0.0.1:8080', timeout=None)

The test succeeds with the patch and fails otherwise.

Currently, one of the test cases still fails, so this cannot be merged. I would like to hear your comments on this, though! Especially from @Zac-HD as the original author of the hypothesis integration and @VladimirWork to see if this patch also solves his problem described in #117.

[seifertm]

As the comment says, return None will cause problems with parametrized hypothesis tests. However, changing return None to yield None breaks test_simple::test_asyncio_marker_fail for no apparent reason. Do you have any idea why this might cause xfail to break?

[Zac-HD]

I can't put my finger on the exact problem, but I have a bad feeling about isolation and the timing of cleanup. That is, that the event_loop might live too long and there might be cases where it gets closed between runs of the test case. The yield-not-return thing makes me super suspicious for this reason.

[seifertm]

You are right: The "yield-not-return" thing is smelly and affects other pytest markers, so I got rid of it. At this point, the parametrization is broken, because the event loop is already closed after the first parameter.

This is weird, because pytest -vv --setup-show -k mark_and_parametrize tests/ shows that the event_loop is correctly set up and torn down before and after each test item. I spent quite some time debuggin, but can't seem to find the issue here…

Thanks to @TimothyFitz who added a test case that reproduces the issue!

[seifertm]

All tests run successfully now and the issue with the prematurely closed event loop seems to be fixed.

I did this by simplifying the wrapper for async functions: pytest_runtest_setup ensures that all tests marked with asyncio require the event_loop fixture and pytest_fixture_setup replaces the current event loop with the loop provided by the fixture. This means if we call get_event_loop(), we can assume that we get the loop provided by the pytest-asyncio fixture. I did exactly that in the async wrapper.

[seifertm]

I also added two refactoring commits that hopefully make the code easier to understand.

[Zac-HD]

Congratulations! This looks great to me, and the refactoring makes it much easier to follow.

[Zac-HD]

Congratulations! This looks great to me, and the refactoring makes it much easier to follow.

[Tinche]

LGTM!