Skip to content

Slow date_histogram after upgrading to 7.3.0 #45702

@brenuart

Description

@brenuart

Elasticsearch version (bin/elasticsearch --version):
Version: 7.3.0, Build: default/rpm/de777fa/2019-07-24T18:30:11.767338Z, JVM: 12.0.1

Plugins installed: []
repository-s3

JVM version (java -version):
JVM shipped with Elasticsearch (rpm)

OS version (uname -a if on a Unix-like system):
Linux es-hot-03.aegaeon-it.com 4.9.120-xxxx-std-ipv6-64 #327490 SMP Thu Aug 16 10:11:35 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
We noticed a severe degradation of date_histogram performance after upgrading from Elasticsearch 6.8.2 to the latest 7.3.0.

The table below shows the execution time of the same query on the same cluster before and after the upgrade. The first execution happens after caches are cleared as follows:

  • ES caches are cleared with POST _cache/clear
  • Filesystem caches are invalidated by doing a cat of all files found in /var/lib/elasticsearch do /dev/null (that's the best we found...)

Timings are as follows:

ES Version   Timings (ms)
------------------------------------
6.8.2        10388,  5797,  7319
7.3.0        21553, 15138, 14833

As you notice 7.3.0 is at least twice slower than 6.8.2.

The dataset is made of about 650m documents spread evenly between 15 indexes. Indexes have 3 shards each and no replica. The cluster is made of three nodes, each with 7Gb RAM (of each 4Gb is allocated to the heap) and 2 vCPUs at 3.1Ghz. There is no activity on the cluster besides this test query.

The query targets only half of the documents (using a date_range) and builds a date_histogram with buckets of 3h. This query is actually what KIbana's Discover panel will do...

{
  "size": 500,
  "sort": [
    {
      "@timestamp": {
        "order": "desc",
        "unmapped_type": "boolean"
      }
    }
  ],
  "aggs": {
    "2": {
      "date_histogram": {
        "field": "@timestamp",
        "interval": "3h",
        "time_zone": "Europe/Berlin",
        "min_doc_count": 1
      }
    }
  },
  "docvalue_fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    }
  ],
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "@timestamp": {
              "format": "strict_date_optional_time",
              "gte": "2019-01-09T00:00:00.000Z",
              "lte": "2019-01-15T00:00:00.000Z"
            }
          }
        }
      ]
    }
  }
}

We are very surprised by this drop of performance...
Did we forgot to change some configurations parameters when doing the upgrade or is it a regression in ES itself ?

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions