using serializer as field fails at import time (e.g. with unit tests) with django 1.7 #1907

robdennis · 2014-09-29T14:04:29Z

I'm using DRF 2.4.2 and due to some custom methods / representations, I'm writing some unit tests for my serializers using py.test and pytest-django, which all worked very well.

I wanted to change a related field (just showed the ID) to the serialized representation so I used an existing serializer as a field. (as explained in the docs

Running tests using py.test (with pytest-django) will blow up with the giant (sanitized) stack trace shown at the bottom of this post.

Consider the documentation example:

class TrackSerializer(serializers.ModelSerializer):
    class Meta:
        model = Track
        fields = ('order', 'title')

class AlbumSerializer(serializers.ModelSerializer):
    tracks = TrackSerializer(many=True)

    class Meta:
        model = Album
        fields = ('album_name', 'artist', 'tracks')

and this key part of the stack trace:

env/lib/python2.7/site-packages/rest_framework/fields.py:146: in __init__
    self.help_text = strip_multiple_choice_msg(smart_text(help_text))
env/lib/python2.7/site-packages/rest_framework/fields.py:116: in strip_multiple_choice_msg
    multiple_choice_msg = force_text(multiple_choice_msg)

The short version based on my investigation:

the instantiation of the TrackSerializer will happen when this is imported
when the TrackSerializer is instantiated, it creates all the fields from the model and sets up help text for them (I assume for the html form).
the help text is using django's force_text method which (check the rest of the trace) will use the "underscore" translation service
this doesn't happen using runserver, which I assume is because something to do with the import timing of loading a WSGI app.

Where I'm at:

I'm not sure if this works using django test runner
I can work around it in my code, so not blocking
I'm not sure what the action is, especially if it works on django test runner
- I was hoping that I could avoid initialization of the serializer by setting the value equal to the class instead of an instance, but no luck, that may be a conceptually simple/ok solution
this may all get thrown out in a pending major release, since I know that serializers were a big focus of the upcoming development calendar

_______________________________________________ ERROR collecting tests/test_serializers.py _______________________________________________
my/app.py: in <module>
    class MySerializer(serializers.ModelSerializer):
hass/endpoint/serializers.py:35: in EndpointSerializer
    related = RelatedSerialzer()
env/lib/python2.7/site-packages/rest_framework/serializers.py:200: in __init__
    self.fields = self.get_fields()
env/lib/python2.7/site-packages/rest_framework/serializers.py:236: in get_fields
    default_fields = self.get_default_fields()
env/lib/python2.7/site-packages/rest_framework/serializers.py:691: in get_default_fields
    serializer_pk_field = self.get_pk_field(pk_field)
env/lib/python2.7/site-packages/rest_framework/serializers.py:818: in get_pk_field
    return self.get_field(model_field)
env/lib/python2.7/site-packages/rest_framework/serializers.py:925: in get_field
    return serializer_field_class(**kwargs)
env/lib/python2.7/site-packages/rest_framework/fields.py:468: in __init__
    super(CharField, self).__init__(*args, **kwargs)
env/lib/python2.7/site-packages/rest_framework/fields.py:272: in __init__
    super(WritableField, self).__init__(source=source, label=label, help_text=help_text)
env/lib/python2.7/site-packages/rest_framework/fields.py:146: in __init__
    self.help_text = strip_multiple_choice_msg(smart_text(help_text))
env/lib/python2.7/site-packages/rest_framework/fields.py:116: in strip_multiple_choice_msg
    multiple_choice_msg = force_text(multiple_choice_msg)
env/lib/python2.7/site-packages/django/utils/encoding.py:85: in force_text
    s = six.text_type(s)
env/lib/python2.7/site-packages/django/utils/functional.py:144: in __text_cast
    return func(*self.__args, **self.__kw)
env/lib/python2.7/site-packages/django/utils/translation/__init__.py:83: in ugettext
    return _trans.ugettext(message)
env/lib/python2.7/site-packages/django/utils/translation/trans_real.py:325: in ugettext
    return do_translate(message, 'ugettext')
env/lib/python2.7/site-packages/django/utils/translation/trans_real.py:306: in do_translate
    _default = translation(settings.LANGUAGE_CODE)
env/lib/python2.7/site-packages/django/utils/translation/trans_real.py:209: in translation
    default_translation = _fetch(settings.LANGUAGE_CODE)
env/lib/python2.7/site-packages/django/utils/translation/trans_real.py:189: in _fetch
    "The translation infrastructure cannot be initialized before the "
E   AppRegistryNotReady: The translation infrastructure cannot be initialized before the apps registry is ready. Check that you don't make non-lazy gettext calls at import time.

The text was updated successfully, but these errors were encountered:

Geekfish · 2014-09-29T14:53:32Z

I have this same issue but it's happening when I'm starting my wsgi server instead.
Tests are running fine, the app is also running fine when ran with runserver.

tomchristie · 2014-09-29T14:55:15Z

@Geekfish Is your case also tracing back to the translation infrastructure, or are you seeing something that's related to AppRegistryNotReady some other ways?

robdennis · 2014-09-29T14:55:35Z

edit: didn't realize I clarified that I was using runserver in the OP, editing in another clarification about the unit test framework I was using

Geekfish · 2014-09-29T15:03:58Z

@tomchristie
The stack trace is very similar:

Traceback (most recent call last):
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 187, in __call__
    response = self.get_response(request)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 199, in get_response
    response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 239, in handle_uncaught_exception
    if resolver.urlconf_module is None:
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/core/urlresolvers.py", line 361, in urlconf_module
    self._urlconf_module = import_module(self.urlconf_name)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/var/www/tt/api/builds/my_project/staging/urls.py", line 6, in <module>
    from my_project.data.legacy_views import (
  File "/var/www/tt/api/builds/my_project/staging/my_project/data/legacy_views.py", line 12, in <module>
    from my_project.data.serializers import SomeObjectSerializer
  File "/var/www/tt/api/builds/my_project/staging/my_project/data/serializers.py", line 10, in <module>
    class SomeOtherObject(BaseSerializer):
  File "/var/www/tt/api/builds/my_project/staging/my_project/data/serializers.py", line 12, in SomeOtherObject
    identifier = fields.CharField(help_text='Unique string identifier.')
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/rest_framework/fields.py", line 470, in __init__
    super(CharField, self).__init__(*args, **kwargs)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/rest_framework/fields.py", line 275, in __init__
    super(WritableField, self).__init__(source=source, label=label, help_text=help_text)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/rest_framework/fields.py", line 142, in __init__
    self.help_text = strip_multiple_choice_msg(smart_text(help_text))
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/rest_framework/fields.py", line 112, in strip_multiple_choice_msg
    multiple_choice_msg = force_text(multiple_choice_msg)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/utils/encoding.py", line 85, in force_text
    s = six.text_type(s)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/utils/functional.py", line 144, in __text_cast
    return func(*self.__args, **self.__kw)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/utils/translation/__init__.py", line 83, in ugettext
    return _trans.ugettext(message)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/utils/translation/trans_real.py", line 325, in ugettext
    return do_translate(message, 'ugettext')
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/utils/translation/trans_real.py", line 306, in do_translate
    _default = translation(settings.LANGUAGE_CODE)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/utils/translation/trans_real.py", line 209, in translation
    default_translation = _fetch(settings.LANGUAGE_CODE)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/utils/translation/trans_real.py", line 189, in _fetch
    "The translation infrastructure cannot be initialized before the "
django.core.exceptions.AppRegistryNotReady: The translation infrastructure cannot be initialized before the apps registry is ready. Check that you don't make non-lazy gettext calls at import time.

Actually I should clarify, the error doesn't happen on startup, it happens when receiving a request.

Geekfish · 2014-10-02T13:27:59Z

Are there any ideas on how to proceed with this one? I think I understand why this is happening, but I'm not sure of what's the best way to fix.
Would checking for the django version before deciding whether to apply the fix in strip_multiple_choice_msg work?

The initial fix could cause problems in Django 1.7 when serializers get used as a field. This is due to `force_text` being applied before full initialization of the apps registry (see encode#1907). The fix doesn't seem to be required after version 1.6: https://code.djangoproject.com/ticket/9321

robdennis · 2014-10-02T18:33:25Z

Thanks for taking on this issue, I appreciate it :)

The initial fix could cause problems in Django 1.7 when serializers get used as a field. This is due to `force_text` being applied before full initialization of the apps registry (see encode#1907). The fix doesn't seem to be required after version 1.6: https://code.djangoproject.com/ticket/9321

carljm · 2014-10-14T00:00:16Z

@Geekfish Does the linked commit fix this for you? I'm curious because (although I think it's a good change nonetheless) it doesn't fix the underlying issue for me - I think this goes deeper.

When I comment out the call to strip_multiple_choice_msg, my traceback just changes to this:

coachhub/tests/converse/test_viewmodels.py:3: in <module>
    from coachhub.converse.viewmodels import ViewThread
coachhub/converse/viewmodels.py:3: in <module>
    from coachhub.api.serializers import MessageSerializer
coachhub/api/serializers.py:108: in <module>
    class CoachProfileSerializer(Serializer):
coachhub/api/serializers.py:109: in CoachProfileSerializer
    user = UserSerializer(required=True)
../../../.venvs/coachhub/lib/python2.7/site-packages/rest_framework/serializers.py:200: in __init__
    self.fields = self.get_fields()
../../../.venvs/coachhub/lib/python2.7/site-packages/rest_framework/serializers.py:236: in get_fields
    default_fields = self.get_default_fields()
../../../.venvs/coachhub/lib/python2.7/site-packages/rest_framework/serializers.py:1095: in get_default_fields
    fields = super(HyperlinkedModelSerializer, self).get_default_fields()
../../../.venvs/coachhub/lib/python2.7/site-packages/rest_framework/serializers.py:749: in get_default_fields
    reverse_rels = opts.get_all_related_objects()
../../../.venvs/coachhub/lib/python2.7/site-packages/django/db/models/options.py:498: in get_all_related_objects
    include_proxy_eq=include_proxy_eq)]
../../../.venvs/coachhub/lib/python2.7/site-packages/django/db/models/options.py:510: in get_all_related_objects_with_model
    self._fill_related_objects_cache()
../../../.venvs/coachhub/lib/python2.7/site-packages/django/db/models/options.py:533: in _fill_related_objects_cache
    for klass in self.apps.get_models(include_auto_created=True):
../../../.venvs/coachhub/lib/python2.7/site-packages/django/utils/lru_cache.py:101: in wrapper
    result = user_function(*args, **kwds)
../../../.venvs/coachhub/lib/python2.7/site-packages/django/apps/registry.py:168: in get_models
    self.check_models_ready()
../../../.venvs/coachhub/lib/python2.7/site-packages/django/apps/registry.py:131: in check_models_ready
    raise AppRegistryNotReady("Models aren't loaded yet.")
E   AppRegistryNotReady: Models aren't loaded yet.

Rather than hitting the problem when it tries to load translations, it just hits a bit later when it tries to load all related objects for a model. (This is with a ModelSerializer).

Basically the issue is that a ModelSerializer does a lot of model introspection when it's instantiated, introspection that really shouldn't be done at import time when not all models are loaded yet, and that simply isn't allowed at import time in Django 1.7. I think fixing this may require non-trivial rethinking of how serializers-as-fields work. Maybe something to be considered in the 3.0 serializer rewrite?

carljm · 2014-10-14T00:01:02Z

(Note: you can work around this by just being careful about where you import your serializers module; you end up doing a lot of imports-within-functions instead of imports at module level.)

tomchristie · 2014-10-14T06:13:37Z

:-|

robdennis · 2014-10-14T14:22:09Z

@carljm I've since run into this in a second project, and importing inside functions is my current work-around.

The cost comes with more friction in adding particular types of tests, but at least I haven't personally seen this new project blow up on requests despite using a serializer as a field (using 2.4.3).

carljm · 2014-10-14T16:10:27Z

In general it doesn't cause a problem on actual requests, as long as you don't import your serializers from a models.py or anything imported from a models.py (in other words, a place that Django actively imports for you at setup time.) Usually that's not the case - your serializers are imported from views or somewhere else that only gets imported on-demand.

tomchristie added the Needs further review label Sep 29, 2014

Geekfish mentioned this issue Oct 2, 2014

Do not apply fix for multichoice help text in later version of Django #1916

Closed

carljm mentioned this issue Oct 15, 2014

Nested Object Serialization in DRF-2.4.3 with "fields" yields crash on startup #1943

Closed

tomchristie added Bug and removed Needs further review labels Oct 16, 2014

tomchristie added this to the 3.0 Release milestone Oct 16, 2014

carljm mentioned this issue Oct 16, 2014

Set up serializer fields lazily on-demand. #1963

Merged

tomchristie closed this as completed in #1963 Oct 31, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

using serializer as field fails at import time (e.g. with unit tests) with django 1.7 #1907

using serializer as field fails at import time (e.g. with unit tests) with django 1.7 #1907

robdennis commented Sep 29, 2014

Geekfish commented Sep 29, 2014

Uh oh!

tomchristie commented Sep 29, 2014

Uh oh!

robdennis commented Sep 29, 2014

Uh oh!

Geekfish commented Sep 29, 2014

Uh oh!

Geekfish commented Oct 2, 2014

Uh oh!

robdennis commented Oct 2, 2014

Uh oh!

carljm commented Oct 14, 2014

Uh oh!

carljm commented Oct 14, 2014

Uh oh!

tomchristie commented Oct 14, 2014

Uh oh!

robdennis commented Oct 14, 2014

Uh oh!

carljm commented Oct 14, 2014

Uh oh!

Uh oh!

using serializer as field fails at import time (e.g. with unit tests) with django 1.7 #1907

using serializer as field fails at import time (e.g. with unit tests) with django 1.7 #1907

Comments

robdennis commented Sep 29, 2014

Geekfish commented Sep 29, 2014

Uh oh!

tomchristie commented Sep 29, 2014

Uh oh!

robdennis commented Sep 29, 2014

Uh oh!

Geekfish commented Sep 29, 2014

Uh oh!

Geekfish commented Oct 2, 2014

Uh oh!

robdennis commented Oct 2, 2014

Uh oh!

carljm commented Oct 14, 2014

Uh oh!

carljm commented Oct 14, 2014

Uh oh!

tomchristie commented Oct 14, 2014

Uh oh!

robdennis commented Oct 14, 2014

Uh oh!

carljm commented Oct 14, 2014

Uh oh!