Using pytest-asyncio fixtures with pytest-twisted functions

[wRAR]

I've initially asked this at https://stackoverflow.com/questions/79451761/using-pytest-twisted-functions-with-pytest-asyncio-fixtures but I now suspect that this is just not possible with the current pytest-twisted code, but maybe it is? If it isn't possible now, can pytest-twisted be improved to support this?

Basically I want to use @pytest_asyncio.fixture fixtures in test functions that use Twisted with --reactor=asyncio. It seems that for this to work correctly all pieces need to use the same asyncio event loop? And there is no way to either create a loop and pass it to both pytest-twisted and pytest-asyncio or extract the loop used by one of these and pass it to another one? If this line of reasoning is correct (and if I understand how do the pieces work), can it be made possible to modify one or both plugins to be able to share the loop/use the installed one?

I also wonder if there are any workarounds in the mean time.

Thank you.

[wRAR]

As a hack I was able to make pytest-asyncio use the existing loop (one created when the reactor is installed) by defining a _session_event_loop() fixture, but the next problem seems to be that while reactor.run(), expectedly, starts the loop with run_forever(), the pytest-asyncio code expects to be able to call run_until_complete() when it wants, which doesn't work.

[altendky]

It seems like a thing that should be doable, though I'll note that I don't work with twisted anymore so it's kind of random whether this grabs my attention vs. other things I might work on.

Could you say some more about the situation that makes this needed? Mostly just fill in some of the details as I think this through (maybe).

Here is where we both decide if we are going to handle a fixture (maybe_mark) and then also where we run it _run_inline_callbacks().

pytest-twisted/src/pytest_twisted/__init__.py

Lines 290 to 306 in ed7f494

	def pytest_fixture_setup(fixturedef, request):
	"""Interface pytest to async for async and async yield fixtures."""
	# TODO: what about _adding_ inlineCallbacks fixture support?
	maybe_mark = _get_mark(fixturedef.func)
	if maybe_mark is None:
	return None
	mark = maybe_mark
	_run_inline_callbacks(
	_async_pytest_fixture_setup,
	fixturedef,
	request,
	mark,
	)
	return not None

Off the top of my head, I think we would need to make that more aware of what test is being run and whether it is a pytest-twisted handled test. I'm not sure if there would be more conditions needed around that like perhaps an option to enable this feature. Maybe.

There would also need to be functionality akin to _run_inline_callbacks() but for the asyncio fixture. Or perhaps an option is added to _run_inline_callbacks() to tell it whether to run the thing in twisted or kick it into asyncio. Maybe https://github.com/twisted/twisted/blob/45dd522fcc218ebbc11afa1ed03c77768e7b8e44/src/twisted/internet/test/test_asyncioreactor.py (or the docs... might make sense) has some reference on doing that.

pytest-twisted/src/pytest_twisted/__init__.py

Lines 333 to 348 in ed7f494

	def _run_inline_callbacks(f, *args):
	"""Interface into Twisted greenlet to run and wait for a deferred."""
	if _instances.gr_twisted is not None:
	if _instances.gr_twisted.dead:
	raise RuntimeError("twisted reactor has stopped")
	def in_reactor(d, f, *args):
	return defer.maybeDeferred(f, *args).chainDeferred(d)
	d = defer.Deferred()
	_instances.reactor.callLater(0.0, in_reactor, d, f, *args)
	blockon_default(d)
	else:
	if not _instances.reactor.running:
	raise RuntimeError("twisted reactor is not running")
	blockingCallFromThread(_instances.reactor, f, *args)

Or, who knows, maybe there's some core issue I haven't thought of yet that makes it all impossible.

[wRAR]

Could you say some more about the situation that makes this needed? Mostly just fill in some of the details as I think this through (maybe).

I have a Scrapy spider (and I need to run it in the test function which requires Twisted and its reactor) and a test website written using aiohttp that I want to scrape with that spider (and I want to run it in-process so I need a fixture that uses the pytest_aiohttp.aiohttp_server fixture to do that). In my view every part of the puzzle makes total sense and isn't replaceable so it would be very nice to be able to combine them (I guess if it's really not possible the pure-asyncio stuff will need to live in a separate process).

There would also need to be functionality akin to _run_inline_callbacks() but for the asyncio fixture.

Oh so you want to run the non-Twisted asyncio parts (so to say) using the pytest-twisted machinery? That makes sense (though I don't know if it's possible). OTOH e.g. pytest_aiohttp.aiohttp_server is decorated with @pytest_asyncio.fixture and so I think needs the pytest-asyncio machinery?

[altendky]

Thanks for filling that in. Yeah, twisted stuff can call asyncio stuff when using the asyncio reactor, just need to use the proper incantation. But also yes, it's got some obvious potential issues.

I guess https://github.com/twisted/twisted/pull/1538/files is actually where I would have been doing that crossover stuff. I don't think the sniffio twisted support ever got done.

Not saying this is the solution, but fyi that afaik you should be able to isolate twisted and asyncio loops in separate threads. Processes shouldn't be required.

[wRAR]

For the record, while moving stuff to a separate thread was harder than I thought, I was able to omit pytest-asyncio and pytest-aiohttp altogether in my project, by copying the pytest-aiohttp.plugin.aiohttp_server fixture into the project and decorating all async fixtures with @pytest_twisted.async_yield_fixture, thus running everything via pytest-twisted. Not sure if this is possible when you don't control all your fixtures.

[altendky]

I'm glad you got to a working state. Let's certainly leave this issue open in case other folks have thoughts or time for it.