Playwright does not play right with asyncio

[Xowap]

Summary

This issue has been previously reported in issues #167, #46, and #29, but has not been fully resolved as the underlying root cause remains unaddressed.

Problem Description

I have observed that the playwright fixture installs a running event loop in the current thread. This behavior leads to subsequent calls to asyncio's run or similar functions becoming unsuccessful. Due to the abstract nature of the Playwright code, I have not yet pinpointed the exact source of the problem.

Additionally, I am unable to reproduce this issue by directly invoking the asyncio API within basic Python scripts not involving Playwright, so I don't really understand the mechanics at play here.

Reproduction

I have created a minimal reproduction repository to demonstrate the issue. Here is a sample test file that highlights the problem:

import asyncio
import pytest


def get_true():
    return True


async def a_get_true():
    return get_true()


@pytest.mark.playwright
def test_pw_true(playwright):
    pass


def test_true():
    assert get_true()


def test_a_true():
    assert asyncio.run(a_get_true())


@pytest.mark.asyncio
async def test_pa_true():
    assert await a_get_true()

In this script, any asynchronous test that follows a test using the playwright fixture results in an error. Changing the test order or skipping the test_pw_true test, which is currently empty, prevents the issue from occurring.

Suggested Solution

One potential solution could be to initialize Playwright in a separate thread or process, with the Pytest API offering a wrapper to handle inter-thread communication. However, this approach might be overly simplistic and may need further refinement.

Temporary Workaround

While awaiting a more permanent and clean solution, I will continue using the hacks mentioned in the above issues. Nonetheless, a robust and canonical fix would be greatly appreciated.

Thank you!

[bartfeenstra]

I have run into this as well. Please add a warning to the documentation, as this means the package is outright incompatible with anything that requires an async test method, because the system under test is asynchronous. It appears that this is the sole reason I cannot use this package and will have to resort to using the JS version of Playwright instead (which comes with the additional overhead of not being able to reuse existing Python fixtures) (not a rant, just an explanation of why this has a bigger impact than it may seem at first).

[mxschmitt]

We are in the process of providing async fixtures - would the following work for you? #74 (comment) - I think the problem is when you mix that pytest-playwright creates its own loop while the pytest-asyncio does as well. They don't work together that well. When going fully sync or fully async things should work.

[bartfeenstra]

I took a quick look at that code and so far it looks good! I'll test is out over the coming days.

[mxschmitt]

Appreciate a ton! Do you have a opinion about betting on asyncio vs. anyio? I saw a community plugin using anyio but not sure how common it is in the testing pytest world.

[bartfeenstra]

I personally only ever use asyncio directly, although some tools I use use AnyIO, but even that is just a wrapper around asyncio and Trio. When authoring libraries, I find it's usually enough to not start one's own loops (unless in a dedicated, synchronous entry point), and allow any API functions to be used through async/await (e.g. API methods return coroutines). So if pytest-playwright(-asyncio) provides asynchronous text fixtures and APIs, then those should work in any testing environment that has a running event loop already (e.g. pytest-asyncio).

[storenth]

+1 asyncio

[mxschmitt]

@storenth do you mind testing #74 (comment) to see if the setup would work for you?

[Xowap]

@mxschmitt thanks for carrying this out, I gave a try to git+https://github.com/microsoft/playwright-pytest#egg=pytest-playwright-asyncio&subdirectory=pytest-playwright-asyncio and it now seems to be working without any modification to my reproduction code, so that's pretty nice I must say

[Xowap]

diff --git a/requirements.txt b/requirements.txt
index 8a6dfc2..2ce0ff9 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -4,8 +4,6 @@ certifi==2024.7.4
     # via requests
 charset-normalizer==3.3.2
     # via requests
-exceptiongroup==1.2.2
-    # via pytest
 greenlet==3.0.3
     # via playwright
 idna==3.7
@@ -15,7 +13,7 @@ iniconfig==2.0.0
 packaging==24.1
     # via pytest
 playwright==1.46.0
-    # via pytest-playwright
+    # via pytest-playwright-asyncio
 pluggy==1.5.0
     # via pytest
 pyee==11.1.0
@@ -25,21 +23,21 @@ pytest==8.3.2
     #   -r requirements.in
     #   pytest-asyncio
     #   pytest-base-url
-    #   pytest-playwright
-pytest-asyncio==0.23.8
-    # via -r requirements.in
+    #   pytest-playwright-asyncio
+pytest-asyncio==0.24.0
+    # via
+    #   -r requirements.in
+    #   pytest-playwright-asyncio
 pytest-base-url==2.1.0
-    # via pytest-playwright
-pytest-playwright==0.5.1
+    # via pytest-playwright-asyncio
+pytest-playwright-asyncio @ git+https://github.com/microsoft/playwright-pytest@fb51327390ccbd3561c1777499934eb88296f1bf#egg=pytest-playwright-asyncio&subdirectory=pytest-playwright-asyncio
     # via -r requirements.in
 python-slugify==8.0.4
-    # via pytest-playwright
+    # via pytest-playwright-asyncio
 requests==2.32.3
     # via pytest-base-url
 text-unidecode==1.3
     # via python-slugify
-tomli==2.0.1
-    # via pytest
 typing-extensions==4.12.2
     # via pyee
 urllib3==2.2.2

To be precise in what changed

[brolnickij]

+1

[mxschmitt]

Awesome thanks for the confirmation @Xowap - I'll close it by that and we'll release pytest-playwright-asyncio soonish. For that I recommend watching #74.

[storenth]

yeah, already done) worked! pls: #74 (comment)

[Xowap]

Amazing, thanks 🙏