Skip to content

using serializer as field fails at import time (e.g. with unit tests) with django 1.7 #1907

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
robdennis opened this issue Sep 29, 2014 · 11 comments · Fixed by #1963
Closed
Labels
Milestone

Comments

@robdennis
Copy link

I'm using DRF 2.4.2 and due to some custom methods / representations, I'm writing some unit tests for my serializers using py.test and pytest-django, which all worked very well.

I wanted to change a related field (just showed the ID) to the serialized representation so I used an existing serializer as a field. (as explained in the docs

Running tests using py.test (with pytest-django) will blow up with the giant (sanitized) stack trace shown at the bottom of this post.

Consider the documentation example:

class TrackSerializer(serializers.ModelSerializer):
    class Meta:
        model = Track
        fields = ('order', 'title')

class AlbumSerializer(serializers.ModelSerializer):
    tracks = TrackSerializer(many=True)

    class Meta:
        model = Album
        fields = ('album_name', 'artist', 'tracks')

and this key part of the stack trace:

env/lib/python2.7/site-packages/rest_framework/fields.py:146: in __init__
    self.help_text = strip_multiple_choice_msg(smart_text(help_text))
env/lib/python2.7/site-packages/rest_framework/fields.py:116: in strip_multiple_choice_msg
    multiple_choice_msg = force_text(multiple_choice_msg)

The short version based on my investigation:

  • the instantiation of the TrackSerializer will happen when this is imported
  • when the TrackSerializer is instantiated, it creates all the fields from the model and sets up help text for them (I assume for the html form).
  • the help text is using django's force_text method which (check the rest of the trace) will use the "underscore" translation service
  • this doesn't happen using runserver, which I assume is because something to do with the import timing of loading a WSGI app.

Where I'm at:

  • I'm not sure if this works using django test runner
  • I can work around it in my code, so not blocking
  • I'm not sure what the action is, especially if it works on django test runner
    • I was hoping that I could avoid initialization of the serializer by setting the value equal to the class instead of an instance, but no luck, that may be a conceptually simple/ok solution
  • this may all get thrown out in a pending major release, since I know that serializers were a big focus of the upcoming development calendar
_______________________________________________ ERROR collecting tests/test_serializers.py _______________________________________________
my/app.py: in <module>
    class MySerializer(serializers.ModelSerializer):
hass/endpoint/serializers.py:35: in EndpointSerializer
    related = RelatedSerialzer()
env/lib/python2.7/site-packages/rest_framework/serializers.py:200: in __init__
    self.fields = self.get_fields()
env/lib/python2.7/site-packages/rest_framework/serializers.py:236: in get_fields
    default_fields = self.get_default_fields()
env/lib/python2.7/site-packages/rest_framework/serializers.py:691: in get_default_fields
    serializer_pk_field = self.get_pk_field(pk_field)
env/lib/python2.7/site-packages/rest_framework/serializers.py:818: in get_pk_field
    return self.get_field(model_field)
env/lib/python2.7/site-packages/rest_framework/serializers.py:925: in get_field
    return serializer_field_class(**kwargs)
env/lib/python2.7/site-packages/rest_framework/fields.py:468: in __init__
    super(CharField, self).__init__(*args, **kwargs)
env/lib/python2.7/site-packages/rest_framework/fields.py:272: in __init__
    super(WritableField, self).__init__(source=source, label=label, help_text=help_text)
env/lib/python2.7/site-packages/rest_framework/fields.py:146: in __init__
    self.help_text = strip_multiple_choice_msg(smart_text(help_text))
env/lib/python2.7/site-packages/rest_framework/fields.py:116: in strip_multiple_choice_msg
    multiple_choice_msg = force_text(multiple_choice_msg)
env/lib/python2.7/site-packages/django/utils/encoding.py:85: in force_text
    s = six.text_type(s)
env/lib/python2.7/site-packages/django/utils/functional.py:144: in __text_cast
    return func(*self.__args, **self.__kw)
env/lib/python2.7/site-packages/django/utils/translation/__init__.py:83: in ugettext
    return _trans.ugettext(message)
env/lib/python2.7/site-packages/django/utils/translation/trans_real.py:325: in ugettext
    return do_translate(message, 'ugettext')
env/lib/python2.7/site-packages/django/utils/translation/trans_real.py:306: in do_translate
    _default = translation(settings.LANGUAGE_CODE)
env/lib/python2.7/site-packages/django/utils/translation/trans_real.py:209: in translation
    default_translation = _fetch(settings.LANGUAGE_CODE)
env/lib/python2.7/site-packages/django/utils/translation/trans_real.py:189: in _fetch
    "The translation infrastructure cannot be initialized before the "
E   AppRegistryNotReady: The translation infrastructure cannot be initialized before the apps registry is ready. Check that you don't make non-lazy gettext calls at import time.
@Geekfish
Copy link
Contributor

I have this same issue but it's happening when I'm starting my wsgi server instead.
Tests are running fine, the app is also running fine when ran with runserver.

@tomchristie
Copy link
Member

@Geekfish Is your case also tracing back to the translation infrastructure, or are you seeing something that's related to AppRegistryNotReady some other ways?

@robdennis
Copy link
Author

edit: didn't realize I clarified that I was using runserver in the OP, editing in another clarification about the unit test framework I was using

@Geekfish
Copy link
Contributor

@tomchristie
The stack trace is very similar:

Traceback (most recent call last):
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 187, in __call__
    response = self.get_response(request)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 199, in get_response
    response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 239, in handle_uncaught_exception
    if resolver.urlconf_module is None:
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/core/urlresolvers.py", line 361, in urlconf_module
    self._urlconf_module = import_module(self.urlconf_name)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/var/www/tt/api/builds/my_project/staging/urls.py", line 6, in <module>
    from my_project.data.legacy_views import (
  File "/var/www/tt/api/builds/my_project/staging/my_project/data/legacy_views.py", line 12, in <module>
    from my_project.data.serializers import SomeObjectSerializer
  File "/var/www/tt/api/builds/my_project/staging/my_project/data/serializers.py", line 10, in <module>
    class SomeOtherObject(BaseSerializer):
  File "/var/www/tt/api/builds/my_project/staging/my_project/data/serializers.py", line 12, in SomeOtherObject
    identifier = fields.CharField(help_text='Unique string identifier.')
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/rest_framework/fields.py", line 470, in __init__
    super(CharField, self).__init__(*args, **kwargs)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/rest_framework/fields.py", line 275, in __init__
    super(WritableField, self).__init__(source=source, label=label, help_text=help_text)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/rest_framework/fields.py", line 142, in __init__
    self.help_text = strip_multiple_choice_msg(smart_text(help_text))
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/rest_framework/fields.py", line 112, in strip_multiple_choice_msg
    multiple_choice_msg = force_text(multiple_choice_msg)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/utils/encoding.py", line 85, in force_text
    s = six.text_type(s)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/utils/functional.py", line 144, in __text_cast
    return func(*self.__args, **self.__kw)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/utils/translation/__init__.py", line 83, in ugettext
    return _trans.ugettext(message)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/utils/translation/trans_real.py", line 325, in ugettext
    return do_translate(message, 'ugettext')
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/utils/translation/trans_real.py", line 306, in do_translate
    _default = translation(settings.LANGUAGE_CODE)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/utils/translation/trans_real.py", line 209, in translation
    default_translation = _fetch(settings.LANGUAGE_CODE)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/utils/translation/trans_real.py", line 189, in _fetch
    "The translation infrastructure cannot be initialized before the "
django.core.exceptions.AppRegistryNotReady: The translation infrastructure cannot be initialized before the apps registry is ready. Check that you don't make non-lazy gettext calls at import time.

Actually I should clarify, the error doesn't happen on startup, it happens when receiving a request.

@Geekfish
Copy link
Contributor

Geekfish commented Oct 2, 2014

Are there any ideas on how to proceed with this one? I think I understand why this is happening, but I'm not sure of what's the best way to fix.
Would checking for the django version before deciding whether to apply the fix in strip_multiple_choice_msg work?

Geekfish added a commit to Geekfish/django-rest-framework that referenced this issue Oct 2, 2014
The initial fix could cause problems in Django 1.7 when serializers
get used as a field. This is due to `force_text` being applied before
full initialization of the apps registry (see encode#1907).

The fix doesn't seem to be required after version 1.6:
https://code.djangoproject.com/ticket/9321
@robdennis
Copy link
Author

Thanks for taking on this issue, I appreciate it :)

Geekfish added a commit to Geekfish/django-rest-framework that referenced this issue Oct 3, 2014
The initial fix could cause problems in Django 1.7 when serializers
get used as a field. This is due to `force_text` being applied before
full initialization of the apps registry (see encode#1907).

The fix doesn't seem to be required after version 1.6:
https://code.djangoproject.com/ticket/9321
Geekfish added a commit to Geekfish/django-rest-framework that referenced this issue Oct 3, 2014
The initial fix could cause problems in Django 1.7 when serializers
get used as a field. This is due to `force_text` being applied before
full initialization of the apps registry (see encode#1907).

The fix doesn't seem to be required after version 1.6:
https://code.djangoproject.com/ticket/9321
@carljm
Copy link
Contributor

carljm commented Oct 14, 2014

@Geekfish Does the linked commit fix this for you? I'm curious because (although I think it's a good change nonetheless) it doesn't fix the underlying issue for me - I think this goes deeper.

When I comment out the call to strip_multiple_choice_msg, my traceback just changes to this:

coachhub/tests/converse/test_viewmodels.py:3: in <module>
    from coachhub.converse.viewmodels import ViewThread
coachhub/converse/viewmodels.py:3: in <module>
    from coachhub.api.serializers import MessageSerializer
coachhub/api/serializers.py:108: in <module>
    class CoachProfileSerializer(Serializer):
coachhub/api/serializers.py:109: in CoachProfileSerializer
    user = UserSerializer(required=True)
../../../.venvs/coachhub/lib/python2.7/site-packages/rest_framework/serializers.py:200: in __init__
    self.fields = self.get_fields()
../../../.venvs/coachhub/lib/python2.7/site-packages/rest_framework/serializers.py:236: in get_fields
    default_fields = self.get_default_fields()
../../../.venvs/coachhub/lib/python2.7/site-packages/rest_framework/serializers.py:1095: in get_default_fields
    fields = super(HyperlinkedModelSerializer, self).get_default_fields()
../../../.venvs/coachhub/lib/python2.7/site-packages/rest_framework/serializers.py:749: in get_default_fields
    reverse_rels = opts.get_all_related_objects()
../../../.venvs/coachhub/lib/python2.7/site-packages/django/db/models/options.py:498: in get_all_related_objects
    include_proxy_eq=include_proxy_eq)]
../../../.venvs/coachhub/lib/python2.7/site-packages/django/db/models/options.py:510: in get_all_related_objects_with_model
    self._fill_related_objects_cache()
../../../.venvs/coachhub/lib/python2.7/site-packages/django/db/models/options.py:533: in _fill_related_objects_cache
    for klass in self.apps.get_models(include_auto_created=True):
../../../.venvs/coachhub/lib/python2.7/site-packages/django/utils/lru_cache.py:101: in wrapper
    result = user_function(*args, **kwds)
../../../.venvs/coachhub/lib/python2.7/site-packages/django/apps/registry.py:168: in get_models
    self.check_models_ready()
../../../.venvs/coachhub/lib/python2.7/site-packages/django/apps/registry.py:131: in check_models_ready
    raise AppRegistryNotReady("Models aren't loaded yet.")
E   AppRegistryNotReady: Models aren't loaded yet.

Rather than hitting the problem when it tries to load translations, it just hits a bit later when it tries to load all related objects for a model. (This is with a ModelSerializer).

Basically the issue is that a ModelSerializer does a lot of model introspection when it's instantiated, introspection that really shouldn't be done at import time when not all models are loaded yet, and that simply isn't allowed at import time in Django 1.7. I think fixing this may require non-trivial rethinking of how serializers-as-fields work. Maybe something to be considered in the 3.0 serializer rewrite?

@carljm
Copy link
Contributor

carljm commented Oct 14, 2014

(Note: you can work around this by just being careful about where you import your serializers module; you end up doing a lot of imports-within-functions instead of imports at module level.)

@tomchristie
Copy link
Member

:-|

@robdennis
Copy link
Author

@carljm I've since run into this in a second project, and importing inside functions is my current work-around.

The cost comes with more friction in adding particular types of tests, but at least I haven't personally seen this new project blow up on requests despite using a serializer as a field (using 2.4.3).

@carljm
Copy link
Contributor

carljm commented Oct 14, 2014

In general it doesn't cause a problem on actual requests, as long as you don't import your serializers from a models.py or anything imported from a models.py (in other words, a place that Django actively imports for you at setup time.) Usually that's not the case - your serializers are imported from views or somewhere else that only gets imported on-demand.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants