Nested Object Serialization in DRF-2.4.3 with "fields" yields crash on startup

[SuperTango]

Hello.

When I use a nested ModelSerializer that uses the "field" parameter to restrict fields, i'm getting a "AttributeError: 'Options' object has no attribute '_related_objects_cache'", and "Models aren't loaded yet" crash upon startup.

Please note i'm using Python 3.4.1, django 1.7, and DRF 2.4.3 (django and DRF are installed via pip) on MacOS 10.9.4

Example:

(model)

class UserProfile(models.Model):
    user = models.OneToOneField(User, unique=True, related_name='userprofile')
    display_name = models.CharField(max_length=100, blank=True, default='')
    user_source = models.CharField(max_length=100)
    remote_id = models.CharField(max_length=200)
    locale = models.CharField(max_length=20)
    timezone = models.CharField(max_length=50, default='America/Los_Angeles')

(serializers)

class UserProfileSerializer(serializers.ModelSerializer):

    class Meta:
        model = UserProfile
        fields = ('display_name', 'locale', 'timezone')

class UserSerializer(serializers.HyperlinkedModelSerializer):
    userprofile = UserProfileSerializer()
    url = serializers.HyperlinkedIdentityField(view_name='user-detail')

    class Meta:
        model = User
        fields = ('email', 'first_name', 'last_name', 'userprofile' )

When launching django, i get:

Traceback (most recent call last):
  File "/Users/tango/python_envs/djangodev1/lib/python3.4/site-packages/django/db/models/options.py", line 508, in get_all_related_objects_with_model
    self._related_objects_cache
AttributeError: 'Options' object has no attribute '_related_objects_cache'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/tango/src/python/serializer_test/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/tango/python_envs/djangodev1/lib/python3.4/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/Users/tango/python_envs/djangodev1/lib/python3.4/site-packages/django/core/management/__init__.py", line 354, in execute
    django.setup()
  File "/Users/tango/python_envs/djangodev1/lib/python3.4/site-packages/django/__init__.py", line 21, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/Users/tango/python_envs/djangodev1/lib/python3.4/site-packages/django/apps/registry.py", line 108, in populate
    app_config.import_models(all_models)
  File "/Users/tango/python_envs/djangodev1/lib/python3.4/site-packages/django/apps/config.py", line 197, in import_models
    self.models_module = import_module(models_module_name)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/importlib/__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "<frozen importlib._bootstrap>", line 1471, in exec_module
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "/Users/tango/src/python/serializer_test/ser_test/models.py", line 38, in <module>
    class UserSerializer(serializers.HyperlinkedModelSerializer):
  File "/Users/tango/src/python/serializer_test/ser_test/models.py", line 39, in UserSerializer
    userprofile = UserProfileSerializer()
  File "/Users/tango/python_envs/djangodev1/lib/python3.4/site-packages/rest_framework/serializers.py", line 200, in __init__
    self.fields = self.get_fields()
  File "/Users/tango/python_envs/djangodev1/lib/python3.4/site-packages/rest_framework/serializers.py", line 236, in get_fields
    default_fields = self.get_default_fields()
  File "/Users/tango/python_envs/djangodev1/lib/python3.4/site-packages/rest_framework/serializers.py", line 749, in get_default_fields
    reverse_rels = opts.get_all_related_objects()
  File "/Users/tango/python_envs/djangodev1/lib/python3.4/site-packages/django/db/models/options.py", line 498, in get_all_related_objects
    include_proxy_eq=include_proxy_eq)]
  File "/Users/tango/python_envs/djangodev1/lib/python3.4/site-packages/django/db/models/options.py", line 510, in get_all_related_objects_with_model
    self._fill_related_objects_cache()
  File "/Users/tango/python_envs/djangodev1/lib/python3.4/site-packages/django/db/models/options.py", line 533, in _fill_related_objects_cache
    for klass in self.apps.get_models(include_auto_created=True):
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/functools.py", line 428, in wrapper
    result = user_function(*args, **kwds)
  File "/Users/tango/python_envs/djangodev1/lib/python3.4/site-packages/django/apps/registry.py", line 168, in get_models
    self.check_models_ready()
  File "/Users/tango/python_envs/djangodev1/lib/python3.4/site-packages/django/apps/registry.py", line 131, in check_models_ready
    raise AppRegistryNotReady("Models aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Models aren't loaded yet.

If i change the UserProfileSerializer to use the not use the "fields" parameter, everything works fine. e.g.

class UserProfileSerializer(serializers.ModelSerializer):

    class Meta:
        model = UserProfile

Also, if i use the "exclude" variable to exclude the fields I don't want instead of using the "fields" parameter to include the fields I want, it works fine. e.g.

class UserProfileSerializer(serializers.ModelSerializer):

    class Meta:
        model = UserProfile
        exclude = ( 'user_source', 'remote_id', 'id', 'user')

This was a big issue for me, but since I found a workaround, it's less urgent now.

[carljm]

This is the same underlying issue as #1907 -- instantiating a serializer at module level (depending on exactly how the serializer is configured) may try to do model introspection work that isn't really safe to do at module import time (because the model system may not be fully setup yet). In previous Django versions this could have silently caused wrong results; in 1.7 it's now an outright error.

The only solution that's come to my mind would be to make using nested serializers a bit more explicit by introducing a NestedSerializerField. So instead of doing this:

class UserSerializer(serializers.HyperlinkedModelSerializer):
    userprofile = UserProfileSerializer(required=True)

You'd do this:

class UserSerializer(serializers.HyperlinkedModelSerializer):
    userprofile = NestedSerializerField(UserProfileSerializer, required=True)

And then internally DRF would lazily instantiate the UserProfileSerializer using the provided kwargs only when it's actually needed. Essentially this would mean removing (or deprecating) the Field interface on Serializers, and moving that work into the NestedSerializerField. To me this seems like a clearer division of responsibilities, although it doesn't make the resulting code quite as simple.

This could be done backwards-compatibly, with the old approach still supported but deprecated. I'd need to do quite a bit more poking around to determine how hard this would be (and how much 3.0 already changes things here; I haven't really looked at the 3.0 code much yet).

@tomchristie what do you think?

[tomchristie]

@carljm - Unsure. Other approach would be to lazily load .fields for ModelSerializer.

[carljm]

@tomchristie That's a great idea; much less invasive. I've filed a pull request implementing it.

I'm not sure how your release process works - I don't see any "stable" 2.x branches. But it would be awfully nice if this fix were backported into a 2.x bugfix release, if 3.0 is going to be a bigger backwards-incompatible release.

(I still kind of like the idea of NestedSerializerField from an architectural clarity perspective; the fact that Serializers inherit Field has always seemed wonky to me. But that's a bigger change, and not necessary to fix this bug.)

[carljm]

Oh nm, I see - it looks like master is the 2.x series, and 3.x has its own branches. So this PR being against master should already get it into the next 2.x release.

[tomchristie]

Now resolved in master.