JSONField does not convert to/from string

[powelleric]

I am inquiring about the behavior of JSONField and specifically about the code here:
https://github.com/encode/django-rest-framework/blob/master/rest_framework/fields.py#L1788-L1804

My expectation for JSONField is that it will take a JSON object and convert it to a string to be stored in the database. In the other direction, it will get the string from the database and convert it to JSON for the response. However, this does not seem to be what is happening.

Looking at the relevant code in to_internal_value, I don't understand why the result of line 1795 (json.dumps) is not returned. That means we fall back to return data on line 1798, which (in the non-binary case) is just the unaltered value. This (python object) is then converted to a string and stored in the DB. However, the string is not json, as it has single quotes around property names. (I can't comment on the binary handling as I don't have experience working with that format.)

Second, I find that to_representation is not returning json. And, in fact, what is stored in the database is invalid for json.loads due to the single quotes.

In my own project, I resolved these issues with the following overrides. I can say that this definitely handles my non-binary use case as I desire but I did not really know how to test the binary handling.

 class JSONNativeField(serializers.JSONField):

     def to_internal_value(self, data):
         if (
            (self.binary or getattr(data, 'is_json_string', False))
             and isinstance(data, bytes)
         ):
             data = data.decode()
         try:
             return json.dumps(data, cls=self.encoder)
         except (TypeError, ValueError):
             self.fail('invalid')

     def to_representation(self, value):
         value = json.loads(value, cls=self.decoder)
         if self.binary:
             value = value.encode()
         return value

My implementation also has the advantage that the POST & GET formats are consistent. This is not the case currently: if I create a new object, the response contains the json field in json format. However, if I perform a get on that same newly-created object, the response contains a string for the json field and it contains single-quoted property names. I think that regardless of the behavior, the response format should be consistent between POST/GET.

Finally, note this SO answer where the person performs their own json.loads in to_representation for their serializer. I would say this person is experiencing the same challenges with JSONField as myself.
https://stackoverflow.com/a/54311320/2117491

[pradeepcep]

I could be wrong, but I don't think the JSONField is intended to return valid JSON strings at all. The documentation only states that the JSONField "validates that the incoming data structure consists of valid JSON primitives".

The current behavior seems to make sense to me because if you had a JSONField in your model, then you are only looking for JSON validation from the serializer. The model field would still expect a Python dict to be passed to it on save. As mentioned in the SO answer you linked, if you had a JSONField in postgres (which is now supported by Django natively with django.db.models.JSONField), then this would be case.

If you are using a backend that doesn't have JSONField support by default, however, and you didn't explicitly convert your value to string using json.dumps(), then you would end up with the string representation of the Python dict, and not a JSON string — which is why you would have seen single quotes in the string stored in the database (assuming you are using a TextField).

If that's the case, then I would recommend using your model layer to abstract your TextField as a JSON field. This would make working with it infinitely easier. For example:

import json
from django.db import models

class UserPreferences(models.Model):
    user = models.ForeignKey('users.User')
    _preferences = models.TextField(db_column='preferences')

    @property
    def preferences(self):
        return self._preferences and json.loads(self._preferences)

    @preferences.setter
    def preferences(self, preferences_dict):
        self._preferences = json.dumps(preferences_dict) if preferences_dict else None

Now if you used a serializer with JSONField for preferences, you would get the behavior you desire:

from rest_framework import serializers
from users.serializers import UserSerializer

class UserPreferencesSerializer(serializers.Serializer):
    user = UserSerializer()
    preferences = serializers.JSONField()

Hope that helps. :)

[powelleric]

Thank you very much for that thorough response.

Now that you have mentioned "if you had a JSONField in your model, then you are only looking for JSON validation from the serializer" I can see the code in a different light. However, that is part of what confused me because I see models.JSONField is new Django 3.1 but I thought that this serializer had been around much longer. What type of model field were people using before Django 3.1?

Regarding the use case "If you are using a backend that doesn't have JSONField support by default", thank you for showing me the technique using the model layer. I did not know that was possible. However, I think that the serializer I proposed still has some merit. If I am understanding the technique you proposed, it requires @property and @preferences.setter for each field, however someone may have several fields of this format, resulting in a lot of repetitive code.

[pradeepcep]

What type of model field were people using before Django 3.1?

Even before 3.1, there was a Postgres-specific JSONField. And before that, I would assume that people did json.dumps() and json.loads() by hand — I am not too sure.

... however someone may have several fields of this format, resulting in a lot of repetitive code.

It's unlikely that with a good database design, there are a lot of JSON fields. However, if my use-case warranted it, I would still add a custom model field and still keep this in the model layer. That way, I can forget about the fact that my database doesn't support JSON fields and assume that whatever value I read from that field is a valid Python dict or None.

But this is subjective, and what has worked for me. If you feel that doing all this in the serializer is better suited to your needs, you should definitely run with it.

[davidgolden]

The behavior I am seeing because of the JSON changes in Django here (django/django@0be51d2) is that when writing to the database, the value returned for a JSONField is a string, where as when only retrieving, the value is correctly formatted as JSON. I would think there should be some logic in to_representation in the JSONField class so that a JSON field would always be returned as JSON, no?

[ReadMost]

You can override to_representation method in serializer in order to get json obj, instead of string.

class ExampleSerializer(serializers.Serializer):
    token_id = serializers.CharField()
    metadata_json = serializers.JSONField()

    def to_representation(self, instance):
        core_repr = super().to_representation(instance)
        core_repr["metadata_json"] = json.loads(core_repr["metadata_json"])
        return core_repr

In this way JSONField validates that string is jsonable and you can return it as json object