model_dump(exclude_defaults=True) fails for numpy fields due to failed equality checks

[coltonbh]

Initial Checks

• I confirm that I'm using Pydantic V2

Description

When I include arbitrary types, like a numpy ndarray, and try to run .model_dump(exclude_defaults=True) the call fails due to an inability to perform equality checks that work with numpy.

Is there a solution for this I'm missing? Perhaps some way to specify __eq__ checks for a field? I'm guessing under the hood pydantic is checking if the field value is set to the default with a == check, which fails for numpy.

Exception:

Traceback (most recent call last):
  File "/home/cbh/dev/hacking/pydantic/example.py", line 26, in <module>
    dumped3 = model.model_dump(exclude_defaults=True)  # Fails!
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cbh/dev/hacking/pydantic/env/lib/python3.11/site-packages/pydantic/main.py", line 309, in model_dump
    return self.__pydantic_serializer__.to_python(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Example Code

from typing import Optional

from numpy import ndarray
from typing_extensions import Annotated
from pydantic import BaseModel, PlainSerializer


NumpyArray = Annotated[ndarray, PlainSerializer(lambda x: x.tolist())]


class MyModel(BaseModel):
    foo: str
    bar: NumpyArray = ndarray([])
    baz: Optional[int] = None

    model_config = {"arbitrary_types_allowed": True}


if __name__ == "__main__":
    model = MyModel(foo="hello", bar=ndarray([1, 2, 3]))

    dumped1 = model.model_dump()  # Works!
    dumped2 = model.model_dump(exclude_none=True)  # Works!
    assert "baz" not in dumped2
    assert "bar" in dumped2

    dumped3 = model.model_dump(exclude_defaults=True)  # Fails!

Python, Pydantic & OS Version

pydantic version: 2.3.0
        pydantic-core version: 2.6.3
          pydantic-core build: profile=release pgo=true
                 install path: /home/cbh/dev/hacking/pydantic/env/lib/python3.11/site-packages/pydantic
               python version: 3.11.5 (main, Aug 31 2023, 10:50:44) [GCC 11.4.0]
                     platform: Linux-5.19.0-41-generic-x86_64-with-glibc2.35
     optional deps. installed: ['typing-extensions']

Selected Assignee: @davidhewitt

[andresliszt]

that comparison is done here: https://github.com/pydantic/pydantic-core/blob/9ec1036b4583d5ada15f7ac3e32fcc04a981746a/src/serializers/fields.rs#L73 --- Is there a chance to instead of compare using python __eq__ can we just check if the default field is not given at init time? I mean if i initialize a model and i set the field that has a default value, i would like to see it in the json/dict even if i initialized it with the default value, because i set it explicitly at init time

import numpy as np
from typing_extensions import Annotated
from pydantic import BaseModel, Field, PlainSerializer, ConfigDict


NumpyArray = Annotated[ndarray, Field(default=np.array([])), PlainSerializer(lambda x: x.tolist())]


class MyModel(BaseModel):
    bar: NumpyArray
    model_config = ConfigDict(arbitrary_types_allowed = True)


if __name__ == "__main__":
    # I would expect an empty list in the serialized object
    model = MyModel(bar=np.array([]))
    model.model_dump(exclude_defaults=True)
    # I would expect {"bar": []}
    model = MyModel()
    # I would expect an empty dictionary {}
    model.model_dump(exclude_defaults=True)

With this approach we do not need to compare default versus given value, we only need to check if the field with default is present

[coltonbh]

Perhaps? But one could set the field later by assignment (i.e., mymodel.bar = np.ndarray([])) and so I think a true comparison at runtime when .model_dumps(exclude_defaults=True) is called may still be necessary.

[andresliszt]

Yes i agree. I think my answer was a bit confusing --- My idea is the following: pydantic models have an attribute model_fields_set that may help to check if the default was set at init or by assignment. Check this Code:

import numpy as np
from typing_extensions import Annotated
from pydantic import BaseModel, Field, PlainSerializer, ConfigDict


NumpyArray = Annotated[np.ndarray, Field(default=np.array([])), PlainSerializer(lambda x: x.tolist())]


class MyModel(BaseModel):
    bar: NumpyArray
    model_config = ConfigDict(arbitrary_types_allowed = True)


if __name__ == "__main__":
    model = MyModel()
    assert model.model_fields_set == set()

    model = MyModel(bar = np.array([]))
    assert model.model_fields_set == {"bar"}
  
    model = MyModel()
    model.bar = np.array([])
    assert model.model_fields_set == {"bar"}

So following my original approach, if bar is set it must be included in the serialized.
but this makes me wonder what is the original idea of ​​the authors about the include_defaults argument, if the idea was exclude always the value even if this was set explicitly then we should have a way to indicate how to compare two objects, maybe in the PlainSerializer?

I feel is not right to have to overwrite __eq__ method in some problematic types such as np.ndarray

[davidhewitt]

if the idea was exclude always the value even if this was set explicitly

I think that it must be this, because there is also exclude_unset=True which already works as you suggest using model_fields_set