Nested objects are empty with elasticseartch-dsl 8.18

[debnet]

When using django-elasticsearch-dsl with latest version of elasticsearch-dsl (8.18) and reindexing my database from scratch, my nested fields are all set with the correct number of items but all empties whereas it was working fine with previous version (8.17.1).

Here my (very-)simplified document:

@registry.register_document
class CompanyDocument(Document):
    type = fields.KeywordField()
    name = fields.TextField(
        analyzer=text_analyzer,
        fields={
            "raw": fields.KeywordField(),
            "suggest": fields.CompletionField(),
        },
    )
    activities = fields.NestedField(
        multi=True,
        attr="startup.all_activities",
        properties=dict(
            id=fields.IntegerField(attr="base_id"),
            name=fields.TextField(),
            desc=fields.TextField(),
        ),
    )

    class Index:
        name = "companies"

    class Django:
        model = Company
        fields = [
            "type",
            "name",
        ]
        queryset_pagination = 1000

    def get_queryset(self):
        return (
            super()
            .get_queryset()
            .prefetch_related(
                Prefetch(
                    "startup__activities",
                    queryset=StartupActivity.objects.annotate(
                        base_id=F("activity_id"),
                        name=F("activity__name"),
                        desc=F("activity__desc"),
                    ).order_by("-weight", "name"),
                    to_attr="all_activities",
                )
            )
        )

When I query a single document, I have the following result:

{
  "_index": "companies",
  "_id": "1",
  "_version": 1,
  "_seq_no": 0,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "type": "startup",
    "name": "My Startup",
    "activities": [
      {},
      {},
      {}
    ]
  }
}

[miguelgrinberg]

Are you using 8.18 in your Elasticsearch server and the elasticsearch-py package as well? Also you mentioned that you have reindexed your data. Are you reporting a problem about data lost during reindex, or data that is not seen by your queries, but is definitely in the index? It is unclear from your description what is the actual problem you are seeing.

[msaraac]

This is not related with this repository. I suggest fix for it. Please check django-es/django-elasticsearch-dsl#494

[debnet]

I'm currently on ElasticSearchv 8.18.2.

Every new document created with nested object is empty.

It works fine with elasticsearch-dsl<8.18 though, even on ES 8.18 cluster.

[miguelgrinberg]

@debnet I believe you 100% that somewhere in there we have a hidden issue, but none of the comments here are helping me figure out or at least reproduce this problem, so I'm completely in the dark about this still. It would be useful if you or other people experiencing this issue can provide actual details that I can use to see this issue here, because once I do, I can debug this and figure it out.

I have written a small example based on your description of the issue at the top, but without using the django package. This does not reproduce the issue:

import os
from elasticsearch_dsl import Document, field, connections

class CompanyDocument(Document):
    type = field.Keyword()
    name = field.Text()
    activities = field.Nested(
        multi=True,
        properties=dict(
            id=field.Integer(),
            name=field.Text(),
        ),
    )

    class Index:
        name = 'companies'

    def __repr__(self):
        return f'type: {self.type}, name: {self.name}, activities: {self.activities}'

connections.create_connection(hosts=[os.environ['ELASTICSEARCH_URL']])

CompanyDocument._index.delete(ignore_unavailable=True)
CompanyDocument.init()

doc = CompanyDocument(type='startup', name='My Startup', activities=[
    {'id': 123, 'name': 'activity_1'},
    {'id': 456, 'name': 'activity_2'},
    {'id': 789, 'name': 'activity_3'},
])
doc.save(refresh=True)

for doc in CompanyDocument.search():
    print(doc)

The output of this script for me is:

type: startup, name: My Startup, activities: [{'id': 123, 'name': 'activity_1'}, {'id': 456, 'name': 'activity_2'}, {'id': 789, 'name': 'activity_3'}]

Can you please modify this script in a way that demonstrates the issue that you have with nested objects?