Running concurrent event loops

[hyansuper]

To test the basic usage of websockets, I want to start a server and test the return string from client, I think server and client should be in separate thread.

The below code passed, but I just want to know if there's a proper/generalized way to do it?

import pytest
import asyncio
import threading
import websockets

async def server(stop_sig):
    async def echo(ws):
        async for msg in ws:
            await ws.send(msg)
    async with websockets.serve(echo, "localhost", 8000):
        await stop_sig

@pytest.fixture
def threaded_server():
    policy = asyncio.get_event_loop_policy()
    loop = policy.new_event_loop()
    sig = asyncio.Future(loop=loop)
    def run_loop(loop, coro):
        loop.run_until_complete(coro)
        loop.close()
    thread = threading.Thread(target=run_loop, args=(loop, server(sig)))
    thread.start()
    yield
    loop.call_soon_threadsafe(sig.set_result, None)
    thread.join()

async def test_client_echo(threaded_server):
    async with websockets.connect('ws://localhost:8000') as ws:
        await ws.send("hello")
        assert await ws.recv() == 'hello'

[seifertm]

I'm not aware of a better solution. asyncio is limited to at most one event loop per thread. This is by design. If you need concurrent event loops, you need to resort to multiple processes or multiple threads like you did.

You might be able to work around that using https://github.com/erdewit/nest_asyncio, but it should only be used for tests.

Does this answer you question?

[hyansuper]

thank you
I think it will be convenient if there’s a decorator to run async function in a separate loop in its own thread, like

@pytest.mark.run_in_threaded_loop
async def test_server():
….

But i don’t know if it’s a very needed feature

[seifertm]

It's worth investigating if this functionality can be added to the new asyncio_event_loop mark (see #620).
However, there should also be a discussion if this should be added.

[wRAR]

Another use case I ran into was using pytest-twisted to run an asyncio loop inside AsyncioSelectorReactor and at the same time using pytest-asyncio to run some asyncio-based fixtures. It doesn't seem possible for them to use separate loops and it doesn't seem possible to tell pytest-asyncio to use the existing loop and don't try to stop it etc.

[seifertm]

Yes, Pytest-asyncio currently assumes that it is the only library managing the asyncio event loop.

@wRAR Can you share more details about your use case? Why is it necessary to use pytest-twisted and pytest-asyncio together?

[wRAR]

@seifertm most of the details are in pytest-dev/pytest-twisted#188, basically I need to run Twisted-based code that uses the asyncio reactor in test functions (which, to my knowledge, requires running the reactor in a correct way, so I either need pytest-twisted or need to run it myself somehow, and in any case I will have a long-running asyncio event loop managed by the reactor) but want to use some asyncio-based fixtures, in my use case they are aiohttp ones, using pytest-aiohttp.plugin.aiohttp_server. I made it work by making pytest-twisted run those fixtures, but that required some copypasting, as normally those require pytest-asyncio, up to using @pytest_asyncio.fixture in pytest-aiohttp (I also hoped to have my fixtures in a separate project, decorated with the same).

Hope this is clear.