Should automatically call django.setup() on django 1.7

[ionelmc]

For now I have this in my conftest.py:

try:
    from django import setup
except ImportError:
    pass
else:
    def pytest_configure():
        setup()

Without it I have these mind-boggling errors (on django 1.7b4):

Traceback (most recent call last):
  File "/home/ionel/osp/django-admin-utils/.tox/3.4-1.7/lib/python3.4/site-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/ionel/osp/django-admin-utils/.tox/3.4-1.7/lib/python3.4/site-packages/django/contrib/admin/sites.py", line 225, in wrapper
    return self.admin_view(view, cacheable)(*args, **kwargs)
  File "/home/ionel/osp/django-admin-utils/.tox/3.4-1.7/lib/python3.4/site-packages/django/utils/decorators.py", line 105, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/home/ionel/osp/django-admin-utils/.tox/3.4-1.7/lib/python3.4/site-packages/django/views/decorators/cache.py", line 52, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/home/ionel/osp/django-admin-utils/.tox/3.4-1.7/lib/python3.4/site-packages/django/contrib/admin/sites.py", line 204, in inner
    return view(request, *args, **kwargs)
  File "/home/ionel/osp/django-admin-utils/.tox/3.4-1.7/lib/python3.4/site-packages/django/views/decorators/cache.py", line 52, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/home/ionel/osp/django-admin-utils/.tox/3.4-1.7/lib/python3.4/site-packages/django/contrib/admin/sites.py", line 401, in index
    'app_url': reverse('admin:app_list', kwargs={'app_label': app_label}, current_app=self.name),
  File "/home/ionel/osp/django-admin-utils/.tox/3.4-1.7/lib/python3.4/site-packages/django/core/urlresolvers.py", line 542, in reverse
    return iri_to_uri(resolver._reverse_with_prefix(view, prefix, *args, **kwargs))
  File "/home/ionel/osp/django-admin-utils/.tox/3.4-1.7/lib/python3.4/site-packages/django/core/urlresolvers.py", line 459, in _reverse_with_prefix
    (lookup_view_s, args, kwargs, len(patterns), patterns))
django.core.urlresolvers.NoReverseMatch: Reverse for 'app_list' with arguments '()' and keyword arguments '{'app_label': 'auth'}' not found. 1 pattern(s) tried: ['admin/(?P<app_label>test_app)/$']

[l0kix2]

Ran into this too today, fixed in conftest also.

[pelme]

Thanks for the report. The issue is a bit odd though, calling django.setup() is already done before any test runs:

https://github.com/pelme/pytest_django/blob/master/pytest_django/plugin.py#L129

Can you provide some additional context where this happens? Please provide the full py.test command output

[maryokhin]

On Django 1.7c1 getting AppRegistryNotReady almost no matter what I do if django.setup() is not called

[paulcollinsiii]

@pelme I just ran into this as well. The cursor wrapper referred to there happens during test running correct? During test discovery for example, if you're importing models and something funny is going on then you can run into some crazy corner cases. I ran into this by having models that use django-simple-history.
Since my tests imported models that used simple history, and simple history reaches into the AppConfig, I was getting AppRegistryNotReady exceptions during test discovery. I'll make a quick PR for where I put a django.setup() that fixed that issue.

This happened on django 1.7c1

[pelme]

There probably are/will be a ton of problems with third party Django apps that does not support Django 1.7 properly with regard to the new app loading.

If AppRegistryNotReady happens during import time (which means that some module reaches into the available apps during import time) - that app does not work well on Django 1.7 at all, and that is a bug with that application.

The solution is to defer any actions that require access to other applications to AppConfig.ready(). I guess a lot of people will notice this when running/collecting tests, but this is just the first sign of a broken third party app or any other code that relies on having the app cache available during import time.

I discussed this briefly with Aymeric Augustin during Djangocon, and that is the advice he gave to the best of my understanding for how to port applications to Django 1.7.

If you are unsure whether a problem with app cache is a bug in your code, a third party app or pytest-django, please provide full tracebacks of the failures and we can discuss each case. Please provide the entire terminal output to make it easier to understand where things go wrong.

[paulcollinsiii]

So here's an example that I'm running into.

py.test --ds datanav.config.settings --dc Testing ./tests --tb native --collect-only 

# long list of tests snipped...

tests/functional/test_form5500/test_admin.py:8: in <module>
    from form5500 import models as f5500_models
src/form5500/models/__init__.py:7: in <module>
    from . import (base, basic, basic_sf, schedule_a, schedule_b, schedule_c, schedule_d,
src/form5500/models/basic.py:306: in <module>
    class Form5500Basic_2006(_Form5500Basic_1999_2008):
/opt/boxen/data/virturalenvs/didj17/lib/python2.7/site-packages/django/db/models/base.py:285: in __new__
    new_class._prepare()
/opt/boxen/data/virturalenvs/didj17/lib/python2.7/site-packages/django/db/models/base.py:343: in _prepare
    signals.class_prepared.send(sender=cls)
/opt/boxen/data/virturalenvs/didj17/lib/python2.7/site-packages/django/dispatch/dispatcher.py:198: in send
    response = receiver(signal=self, sender=sender, **named)
/opt/boxen/data/virturalenvs/didj17/lib/python2.7/site-packages/simple_history/models.py:88: in finalize
    history_model = self.create_history_model(sender)
/opt/boxen/data/virturalenvs/didj17/lib/python2.7/site-packages/simple_history/models.py:120: in create_history_model
    app = apps.app_configs[model._meta.app_label]
E   KeyError: 'form5500'

so ya, simple_history is messing with the app_configs dictionary but during normal use it ends up working because models aren't imported until after AppConfig has been setup. In https://github.com/pelme/pytest_django/blob/master/pytest_django/plugin.py#L76
if I add

from .compat import setup
setup()

then that particular set of errors goes away (I'm fighting a thing with FactoryBoy as well...)
Of course by adding those lines I cause all kinds of test failures in your tests 😢

[pelme]

Hmm. It looks like django-simple-history is intentionally hooking into the app config even when it is not ready, to be able to dynamically create the history model.

It also looks like django-simple-history is using some kind of heuristic to find the proper module for the models. This might be the problem in this case since you are not using a plain models.py.

I am not familiar with django-simple-history internals, but it uses internal Django API:s and uses a lot of meta things. This might be an edge case that breaks with how it detects models, but I think it can possibly be fixed by moving some parts of the initialization to AppConfig.ready().

This bug might be an error of implicit module loading (the very error that the Django 1.7 app loading should fix). This may work when invoked from runserver just because of the import's happens in another order then, which does not trigger the error.

What happens if you just run a python shell (not manage.py shell, just python) and run "from form5500 import models as f5500_models" ? Does that work?

Could you open an issue on django-simple-history to see if they think this is a bug?

[pelme]

(Btw, just running setup() whenever pytest-django is loaded will not work. pytest-django is very careful to not load Django when Django is not imported, and this would just break that feature. setup() is called properly, but after test collection, but always before a test runs)

[paulcollinsiii]

Oh, darn. I didn't realize that was a very deliberate feature. I'll close my PR then since it's completely the wrong solution in that case. On that feature though, doesn't the normal django test running fully load settings then do test discovery?

As for importing those models through the normal shell I get
django.core.exceptions.ImproperlyConfigured: Requested setting AUTH_USER_MODEL, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings. If I define the django settings module, then django-configurations squawks. I have a feeling that loading that model any way other than manage.py shell will cause 🔥

I guess the end result for my tests is just update my conftest.py module to force loading of django settings before test discovery? Or should I update my PR to make that an option? I have a feeling that during the whole 1.7 upgrade there are going to be a lot of third party modules that have these kinds of issues.

[blueyed]

@pelme

I think it can possibly be fixed by moving some parts of the initialization to AppConfig.ready()

Do you mean that this is something that pytest-django could do?

@paulcollinsiii
I also think that filing an issue with django-simple-history would help to address, discuss and fix this in general.

[pelme]

@blueyed No, I mean that django-simple-history should probably move any initialization that depends on having app config ready from import time to its AppConfig.ready().

I left a comment in the django-simple-history issue too.

I think that the order of things should be

• Import things
• Run django.setup() (which in turn triggers all AppConfig.ready() triggers, with a fully reliable and populated app config)
• Run anything else (test, webservers, celery workers, ...)

The "Import things" step can be omitted when running a web server for instance, since you do not really care about the modules, and django.setup() will import all things needed.

This is currently how it works in pytest-django. If anyone else is having 1.7 issues, please post test outputs and we can try to debug/discuss them to find the root cause.