Support for hypothesis in async methods

[carver]

hypothesis exits early when used on pytest-asyncio tests, complaining that the test returns a coroutine instead of None. This issue talks about how trio added support for hypothesis: HypothesisWorks/hypothesis#968

I can work around this with a decorator:

def run_in_event_loop(async_func):
    @functools.wraps(async_func)
    def wrapped(operations, queue_size, add_size, get_size, event_loop):
        event_loop.run_until_complete(asyncio.ensure_future(
            async_func(operations, queue_size, add_size, get_size, event_loop),
            loop=event_loop,
        ))
    return wrapped

@hypothesis.given(...)
@run_in_event_loop
@pytest.mark.asyncio
async def test_...

But it looks like the recommended approach is to tinker with the hypothesis attribute added to the test, like: https://github.com/python-trio/pytest-trio/pull/44/files#diff-aacce3c1178656a02aecc59c94209433R231

Is this something pytest-asyncio is interested in supporting?

[japrogramer]

what is the status of this, I would like to use hypothesis for my async project so my tests must also be async ... and don't want to use trio.

I was thinking of doing something like this

@given(st.lists(
  st.floats(allow_nan=False, allow_infinity=False), min_size=1))
def test_mean(xs):
    old_loop = asyncio.get_event_loop()
    new_loop = asyncio.new_event_loop()
    try:
        asyncio.set_event_loop(new_loop)
        new_loop.run_until_complete(my_corrutine(xs))
    except Exception as e:
         raise e
    finally:
        asyncio.set_event_loop(old_loop)

[japrogramer]

turned it into a fixture

@pytest.fixture(scope="session")
def event_loop():
    """
    Run coroutines in non async code
    by using a new event_loop
    """
    old_loop = asyncio.get_event_loop()
    new_loop = asyncio.new_event_loop()
    asyncio.set_event_loop(new_loop)
    try:
        yield new_loop
    finally:
        asyncio.set_event_loop(old_loop)

[nicoddemus]

Just a quick remark: the try/finally block is not required for pytest fixtures, the code after the yield will always be executed regardless if the test raised an exception or not (contrary to contextlib.contextmanager).

[japrogramer]

neat, makes sense .. i copy pasted the code into a fixture, thats why i kept it like that

[Zac-HD]

Hi all, maintainer of Hypothesis + Trio + Pytest here - I wrote the linked integration and would be happy to have a go here. Only reason I hadn't already added this was that I don't use asyncio much myself and hoped someone else might take on the task 😄

FYI @japrogramer the integration is basically equivalent to your fixture / @carver's decorator, automatically applied to all tests with pytest.mark.trio that also use Hypothesis. It might be a bit more complicated for asyncio because of explicit event loops, but we'll see I guess!

[Zac-HD]

OK, a first attempt is up as #102. @carver @japrogramer - can you check that this works for you?

[carver]

OK, a first attempt is up as #102. @carver @japrogramer - can you check that this works for you?

Yes, the change does appear to work on the hypothesis tests, when I pip install -e from the Zac-HD/hypothesis checkout (at d4c1b92).

However... my other pytest.mark.asyncio tests start emitting a bunch of warnings like:

tests/trinity/utils/test_task_queue.py::test_queue_size_reset_after_complete
  /home/jcarver/code/py-evm/venv/lib/python3.6/site-packages/_pytest/python.py:195: RuntimeWarning: coroutine 'test_queue_size_reset_after_complete' was never awaited
    testfunction(**testargs)

(they are all green with pytest-asyncio v0.9.0)

I haven't had a chance to dig into why, but maybe it is unrelated to your change.

[japrogramer]

@carver that error means that a coroutine was never awaited but called as if it where.

I merged the pull request into master and this is what i got
@Zac-HD

▶ coverage run -m pytest                                                                                                                                                                                 ∞
=============================================================================================== test session starts ================================================================================================
platform linux -- Python 3.7.1, pytest-4.0.1, py-1.7.0, pluggy-0.8.0
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/archangel/VENV/pytest/pytest-asyncio/.hypothesis/examples')
rootdir: /home/archangel/VENV/pytest/pytest-asyncio, inifile: setup.cfg
plugins: asyncio-0.10.0.dev0, hypothesis-3.83.0
collected 38 items

tests/test_dependent_fixtures.py .                                                                                                                                                                           [  2%]
tests/test_event_loop_scope_35.py ...                                                                                                                                                                        [ 10%]
tests/test_hypothesis_integration.py ....                                                                                                                                                                    [ 21%]
tests/test_simple.py ..x......                                                                                                                                                                               [ 44%]
tests/test_simple_35.py ......                                                                                                                                                                               [ 60%]
tests/test_subprocess.py ..                                                                                                                                                                                  [ 65%]
tests/async_fixtures/test_async_fixtures_35.py .                                                                                                                                                             [ 68%]
tests/async_fixtures/test_async_fixtures_scope.py .                                                                                                                                                          [ 71%]
tests/async_fixtures/test_async_gen_fixtures_35.py ..                                                                                                                                                        [ 76%]
tests/async_fixtures/test_async_gen_fixtures_36.py ..                                                                                                                                                        [ 81%]
tests/async_fixtures/test_coroutine_fixtures.py s                                                                                                                                                            [ 84%]
tests/async_fixtures/test_nested_36.py .                                                                                                                                                                     [ 86%]
tests/markers/test_class_marker_35.py .                                                                                                                                                                      [ 89%]
tests/markers/test_module_marker_35.py ..                                                                                                                                                                    [ 94%]
tests/multiloop/test_alternative_loops.py ..                                                                                                                                                                 [100%]
============================================================================================= short test summary info ==============================================================================================
SKIP [1] tests/async_fixtures/test_coroutine_fixtures.py:23: @asyncio.coroutine fixtures are not supported yet
XFAIL tests/test_simple.py::test_asyncio_marker_fail
  need a failure

================================================================================= 36 passed, 1 skipped, 1 xfailed in 1.05 seconds ==================================================================================

[carver]

@carver that error means that a coroutine was never awaited but called as if it where.

Right, but the method that isn't being awaited is the whole test. pytest.mark.asyncio is supposed to await it for you. Also, the tests pass without warning in pytest-asyncio v0.9.0.

[Zac-HD]

It looks like my detection to avoid wrapping Hypothesis tests twice was a bit too loose - try again with 1b66314?

[carver]

Yup, all green!

[japrogramer]

@Zac-HD not for me .. hmm, am i running this correctly?

▶ coverage run -m pytest                                                                                                                                                                               86%
=============================================================================================== test session starts ================================================================================================
platform linux -- Python 3.7.1, pytest-4.0.1, py-1.7.0, pluggy-0.8.0
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/archangel/VENV/pytest/pytest-asyncio/.hypothesis/examples')
rootdir: /home/archangel/VENV/pytest/pytest-asyncio, inifile: setup.cfg
plugins: asyncio-0.10.0.dev0, hypothesis-3.83.0
collected 38 items

tests/test_dependent_fixtures.py .                                                                                                                                                                           [  2%]
tests/test_event_loop_scope_35.py ...                                                                                                                                                                        [ 10%]
tests/test_hypothesis_integration.py ....                                                                                                                                                                    [ 21%]
tests/test_simple.py ..x......                                                                                                                                                                               [ 44%]
tests/test_simple_35.py ......                                                                                                                                                                               [ 60%]
tests/test_subprocess.py ..                                                                                                                                                                                  [ 65%]
tests/async_fixtures/test_async_fixtures_35.py .                                                                                                                                                             [ 68%]
tests/async_fixtures/test_async_fixtures_scope.py .                                                                                                                                                          [ 71%]
tests/async_fixtures/test_async_gen_fixtures_35.py ..                                                                                                                                                        [ 76%]
tests/async_fixtures/test_async_gen_fixtures_36.py ..                                                                                                                                                        [ 81%]
tests/async_fixtures/test_coroutine_fixtures.py s                                                                                                                                                            [ 84%]
tests/async_fixtures/test_nested_36.py .                                                                                                                                                                     [ 86%]
tests/markers/test_class_marker_35.py .                                                                                                                                                                      [ 89%]
tests/markers/test_module_marker_35.py ..                                                                                                                                                                    [ 94%]
tests/multiloop/test_alternative_loops.py ..                                                                                                                                                                 [100%]
============================================================================================= short test summary info ==============================================================================================
SKIP [1] tests/async_fixtures/test_coroutine_fixtures.py:23: @asyncio.coroutine fixtures are not supported yet
XFAIL tests/test_simple.py::test_asyncio_marker_fail
  need a failure

================================================================================= 36 passed, 1 skipped, 1 xfailed in 1.58 seconds ==================================================================================

and here is the diff from master

diff --git a/.gitignore b/.gitignore
index a49cb17..0975808 100644
--- a/.gitignore
+++ b/.gitignore
@@ -22,6 +22,7 @@ var/
 *.egg-info/
 .installed.cfg
 *.egg
+.hypothesis/
 
 # PyInstaller
 #  Usually these files are written by a python script from a template
diff --git a/README.rst b/README.rst
index 8c36da8..69ace41 100644
--- a/README.rst
+++ b/README.rst
@@ -178,6 +178,9 @@ Changelog
 
 0.10.0. (UNRELEASED)
 ~~~~~~~~~~~~~~~~~~~~
+- ``pytest-asyncio`` integrates with `Hypothesis <https://hypothesis.readthedocs.io>`_
+  to support ``@given`` on async test functions using ``asyncio``.
+  `#102` <https://github.com/pytest-dev/pytest-asyncio/pull/102>
 
 0.9.0 (2018-07-28)
 ~~~~~~~~~~~~~~~~~~
diff --git a/pytest_asyncio/plugin.py b/pytest_asyncio/plugin.py
index 584efbf..26a37d4 100644
--- a/pytest_asyncio/plugin.py
+++ b/pytest_asyncio/plugin.py
@@ -1,6 +1,7 @@
 """pytest-asyncio implementation."""
 import asyncio
 import contextlib
+import functools
 import inspect
 import socket
 
@@ -139,7 +140,8 @@ def pytest_pyfunc_call(pyfuncitem):
     function call.
     """
     for marker_name, fixture_name in _markers_2_fixtures.items():
-        if marker_name in pyfuncitem.keywords:
+        if marker_name in pyfuncitem.keywords \
+                and not getattr(pyfuncitem.obj, 'is_hypothesis_test', False):
             event_loop = pyfuncitem.funcargs[fixture_name]
 
             funcargs = pyfuncitem.funcargs
@@ -152,11 +154,39 @@ def pytest_pyfunc_call(pyfuncitem):
             return True
 
 
+def wrap_in_sync(func):
+    """Return a sync wrapper around an async function."""
+
+    @functools.wraps(func)
+    def inner(**kwargs):
+        loop = asyncio.get_event_loop_policy().new_event_loop()
+        try:
+            coro = func(**kwargs)
+            if coro is not None:
+                future = asyncio.ensure_future(coro, loop=loop)
+                loop.run_until_complete(future)
+        finally:
+            loop.close()
+
+    return inner
+
+
 def pytest_runtest_setup(item):
     for marker, fixture in _markers_2_fixtures.items():
         if marker in item.keywords and fixture not in item.fixturenames:
             # inject an event loop fixture for all async tests
             item.fixturenames.append(fixture)
+    if item.get_closest_marker("asyncio") is not None:
+        if hasattr(item.obj, 'hypothesis'):
+            # If it's a Hypothesis test, we insert the wrap_in_sync decorator
+            item.obj.hypothesis.inner_test = wrap_in_sync(
+                item.obj.hypothesis.inner_test
+            )
+        elif getattr(item.obj, 'is_hypothesis_test', False):
+            pytest.fail(
+                'test function `%r` is using Hypothesis, but pytest-asyncio '
+                'only works with Hypothesis 3.64.0 or later.' % item
+            )
 
 
 # maps marker to the name of the event loop fixture that will be available
diff --git a/setup.py b/setup.py
index 2c26c42..97ef460 100644
--- a/setup.py
+++ b/setup.py
@@ -43,7 +43,11 @@ setup(
     install_requires=["pytest >= 3.0.6"],
     extras_require={
         ':python_version == "3.5"': "async_generator >= 1.3",
-        "testing": ["coverage", "async_generator >= 1.3"],
+        "testing": [
+            "coverage",
+            "async_generator >= 1.3",
+            "hypothesis >= 3.64",
+        ],
     },
     entry_points={"pytest11": ["asyncio = pytest_asyncio.plugin"]},
 )
diff --git a/tests/test_hypothesis_integration.py b/tests/test_hypothesis_integration.py
new file mode 100644
index 0000000..562f477
--- /dev/null
+++ b/tests/test_hypothesis_integration.py
@@ -0,0 +1,27 @@
+"""Tests for the Hypothesis integration, which wraps async functions in a
+sync shim for Hypothesis.
+"""
+
+import pytest
+
+from hypothesis import given, strategies as st
+
+
+@given(st.integers())
[email protected]
+async def test_mark_inner(n):
+    assert isinstance(n, int)
+
+
[email protected]
+@given(st.integers())
+async def test_mark_outer(n):
+    assert isinstance(n, int)
+
+
[email protected]("y", [1, 2])
+@given(x=st.none())
[email protected]
+async def test_mark_and_parametrize(x, y):
+    assert x is None
+    assert y in (1, 2)

pip freeze

async-generator==1.10
atomicwrites==1.2.1
attrs==18.2.0
coverage==4.5.2
hypothesis==3.83.0
more-itertools==4.3.0
pluggy==0.8.0
py==1.7.0
pytest==4.0.1
-e git+https://github.com/pytest-dev/pytest-asyncio.git@923fe9a95392148fb73985ac0e5a47b4e4d18098#egg=pytest_asyncio
six==1.12.0

[Zac-HD]

Hmm, that's the correct diff from master at least. One skip and one xfail might not actually be a problem though - can you check if they occur on master?

[japrogramer]

thats weird i uninstalled and git reset --hard to be on master again
and i have the same errror

▶ pip uninstall pytest_asyncio                                                                                                                                                                               ∞
Uninstalling pytest-asyncio-0.10.0.dev0:
  Would remove:
    /home/archangel/VENV/pytest/lib/python3.7/site-packages/pytest-asyncio.egg-link
Proceed (y/n)? y
  Successfully uninstalled pytest-asyncio-0.10.0.dev0
(pytest)
VENV/pytest/pytest-asyncio  master ✔                                                                                                                                                                          72d
▶ pip install -e .                                                                                                                                                                                           ∞
Obtaining file:///home/archangel/VENV/pytest/pytest-asyncio
Requirement already satisfied: pytest>=3.0.6 in /home/archangel/VENV/pytest/lib/python3.7/site-packages (from pytest-asyncio==0.10.0.dev0) (4.0.1)
Requirement already satisfied: pluggy>=0.7 in /home/archangel/VENV/pytest/lib/python3.7/site-packages (from pytest>=3.0.6->pytest-asyncio==0.10.0.dev0) (0.8.0)
Requirement already satisfied: attrs>=17.4.0 in /home/archangel/VENV/pytest/lib/python3.7/site-packages (from pytest>=3.0.6->pytest-asyncio==0.10.0.dev0) (18.2.0)
Requirement already satisfied: more-itertools>=4.0.0 in /home/archangel/VENV/pytest/lib/python3.7/site-packages (from pytest>=3.0.6->pytest-asyncio==0.10.0.dev0) (4.3.0)
Requirement already satisfied: atomicwrites>=1.0 in /home/archangel/VENV/pytest/lib/python3.7/site-packages (from pytest>=3.0.6->pytest-asyncio==0.10.0.dev0) (1.2.1)
Requirement already satisfied: six>=1.10.0 in /home/archangel/VENV/pytest/lib/python3.7/site-packages (from pytest>=3.0.6->pytest-asyncio==0.10.0.dev0) (1.12.0)
Requirement already satisfied: setuptools in /home/archangel/VENV/pytest/lib/python3.7/site-packages (from pytest>=3.0.6->pytest-asyncio==0.10.0.dev0) (39.0.1)
Requirement already satisfied: py>=1.5.0 in /home/archangel/VENV/pytest/lib/python3.7/site-packages (from pytest>=3.0.6->pytest-asyncio==0.10.0.dev0) (1.7.0)
Installing collected packages: pytest-asyncio
  Running setup.py develop for pytest-asyncio
Successfully installed pytest-asyncio
(pytest)
VENV/pytest/pytest-asyncio  master ✔                                                                                                                                                                          72d
▶ git status                                                                                                                                                                                                ∞
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean
(pytest)
VENV/pytest/pytest-asyncio  master ✔                                                                                                                                                                          72d
▶ pip freeze                                                                                                                                                                                                 ∞
async-generator==1.10
atomicwrites==1.2.1
attrs==18.2.0
coverage==4.5.2
hypothesis==3.83.0
more-itertools==4.3.0
pluggy==0.8.0
py==1.7.0
pytest==4.0.1
-e git+https://github.com/pytest-dev/pytest-asyncio.git@923fe9a95392148fb73985ac0e5a47b4e4d18098#egg=pytest_asyncio
six==1.12.0


▶ coverage run -m pytest                                                                                                                                                                                     ∞
=============================================================================================== test session starts ================================================================================================
platform linux -- Python 3.7.1, pytest-4.0.1, py-1.7.0, pluggy-0.8.0
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/archangel/VENV/pytest/pytest-asyncio/.hypothesis/examples')
rootdir: /home/archangel/VENV/pytest/pytest-asyncio, inifile: setup.cfg
plugins: asyncio-0.10.0.dev0, hypothesis-3.83.0
collected 34 items

tests/test_dependent_fixtures.py .                                                                                                                                                                           [  2%]
tests/test_event_loop_scope_35.py ...                                                                                                                                                                        [ 11%]
tests/test_simple.py ..x......                                                                                                                                                                               [ 38%]
tests/test_simple_35.py ......                                                                                                                                                                               [ 55%]
tests/test_subprocess.py ..                                                                                                                                                                                  [ 61%]
tests/async_fixtures/test_async_fixtures_35.py .                                                                                                                                                             [ 64%]
tests/async_fixtures/test_async_fixtures_scope.py .                                                                                                                                                          [ 67%]
tests/async_fixtures/test_async_gen_fixtures_35.py ..                                                                                                                                                        [ 73%]
tests/async_fixtures/test_async_gen_fixtures_36.py ..                                                                                                                                                        [ 79%]
tests/async_fixtures/test_coroutine_fixtures.py s                                                                                                                                                            [ 82%]
tests/async_fixtures/test_nested_36.py .                                                                                                                                                                     [ 85%]
tests/markers/test_class_marker_35.py .                                                                                                                                                                      [ 88%]
tests/markers/test_module_marker_35.py ..                                                                                                                                                                    [ 94%]
tests/multiloop/test_alternative_loops.py ..                                                                                                                                                                 [100%]
Exception ignored in: <function BaseEventLoop.__del__ at 0x7f6cdb9ad620>
Traceback (most recent call last):
  File "/usr/lib/python3.7/asyncio/base_events.py", line 612, in __del__
    source=self)
ResourceWarning: unclosed event loop <_UnixSelectorEventLoop running=False closed=False debug=False>
Exception ignored in: <socket.socket fd=10, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>
ResourceWarning: unclosed <socket.socket fd=10, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>
Exception ignored in: <socket.socket fd=11, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>
ResourceWarning: unclosed <socket.socket fd=11, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>
============================================================================================= short test summary info ==============================================================================================
SKIP [1] tests/async_fixtures/test_coroutine_fixtures.py:23: @asyncio.coroutine fixtures are not supported yet
XFAIL tests/test_simple.py::test_asyncio_marker_fail
  need a failure

================================================================================= 32 passed, 1 skipped, 1 xfailed in 1.22 seconds ==================================================================================

[Zac-HD]

OK, I've just gone and checked - neither of those are problems and both are present on master, so it looks like my patch does work 🎉

[gjcarneiro]

I tried the patch and it seems to work. I have to deal with "future attached to a different loop" problems, but interaction with hypothesis seems to work at least.

[gjcarneiro]

Although, I suspect that the "attached to a different loop" problems I'm having may be due to this patch, since it does loop = asyncio.get_event_loop_policy().new_event_loop().

Yes, things will fail because wrap_in_sync() creates one asyncio event loop, and the event_loop() fixture creates another event loop.

By changing loop = asyncio.get_event_loop_policy().new_event_loop() to loop = asyncio.get_event_loop() (and removing the loop.close()), it seems to work. Though probably not 100% correct, we should get the event_loop fixture and use that, but I don't know how to do that.