Sync/Async issues when integrating with Django 3.1.5

[caramdache]

I'm currently converting some puppeteer code to playright-python. And I'm taking this opportunity to use my django models rather than raw SQL queries:

#!/usr/bin/env python3

import os, sys
import django

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mysite.settings")
django.setup()

from playwright import sync_playwright

from models import SomeModelClass

with sync_playwright() as playwright:
    objs = SomeModelClass.objects.filter(pk=1)
    print(objs)

However, I'm running into some issues and I would appreciate some guidance:

1. I started by using the sync API, but when I try to run a query like: SomeModelClass.objects.get(pk=1), I get the following error: django.core.exceptions.SynchronousOnlyOperation: You cannot call this from an async context - use a thread or sync_to_async.
2. I then tried to use sync_to_async, from https://docs.djangoproject.com/en/3.1/topics/async/#asgiref.sync.sync_to_async. But this leads to further errors.
3. I then converted by entire code to use the async API, but this did not quite solve the issue, since django querysets are evaluated lazyly, i.e. outside the (in this case) async method.
4. I did build upon 3. and forced the queryset to be evaluated inside the async method:

@sync_to_async
def getData():
    objs= SomeModelClass.objects.filter(pk=1)
    return list(objs)

This seemed to work, but at the expense of code complexity and I starting running into timeout issues.

5. The only solution that's been easy to use was as follows:

os.environ["DJANGO_ALLOW_ASYNC_UNSAFE"] = "true"

But, as pointed out by django, it may result in data loss.

Any advice from the team?

[kumaraditya303]

I'm currently converting some puppeteer code to playright-python. And I'm taking this opportunity to use my django models rather than raw SQL queries:

#!/usr/bin/env python3

import os, sys
import django

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mysite.settings")
django.setup()

from playwright import sync_playwright

from models import SomeModelClass

with sync_playwright() as playwright:
    objs = SomeModelClass.objects.filter(pk=1)
    print(objs)

However, I'm running into some issues and I would appreciate some guidance:

1. I started by using the sync API, but when I try to run a query like: SomeModelClass.objects.get(pk=1), I get the following error: django.core.exceptions.SynchronousOnlyOperation: You cannot call this from an async context - use a thread or sync_to_async.
2. I then tried to use sync_to_async, from https://docs.djangoproject.com/en/3.1/topics/async/#asgiref.sync.sync_to_async. But this leads to further errors.
3. I then converted by entire code to use the async API, but this did not quite solve the issue, since django querysets are evaluated lazyly, i.e. outside the (in this case) async method.
4. I did build upon 3. and forced the queryset to be evaluated inside the async method:

@sync_to_async
def getData():
    objs= SomeModelClass.objects.filter(pk=1)
    return list(objs)

This seemed to work, but at the expense of code complexity and I starting running into timeout issues.

1. The only solution that's been easy to use was as follows:

os.environ["DJANGO_ALLOW_ASYNC_UNSAFE"] = "true"

But, as pointed out by django, it may result in data loss.

Any advice from the team?

The documentation is pointing about not using it in the application, not in testing, in testing it is fine also the env var should be configured in a conftest.py file to separate it from the real application. Ref

[caramdache]

os.environ["DJANGO_ALLOW_ASYNC_UNSAFE"] = "true"

But, as pointed out by django, it may result in data loss.
Any advice from the team?

The documentation is pointing about not using it in the application, not in testing, in testing it is fine also the env var should be configured in a conftest.py file to separate it from the real application. Ref

But I am using it in an application, not in testing. So this is why using DJANGO_ALLOW_ASYNC_UNSAFE is not such a good idea. Sorry if that wasn't clear from my original post.

[pavelfeldman]

@caramdache thanks for the report.

It looks like the SQL business is annotated with the async_unsafe decoration implemented here. Every call annotated as such will check if it is running inside event loop.

Playwright's sync API uses asyncio loop internally, it uses greenlets to simulate blocking synchronous behavior, but still runs the event loop inside. So django is getting confused. I'll see if we can do something about it. But the nature of the check suggests that it is a development-time assertion. If you know that you only hit this from within Playwright, it is probably fine to disable the check. Having said that, if you disable it and bring in more code into your project you might miss a real offender.

[mxschmitt]

Closing for now since the workaround seems working.