-
Notifications
You must be signed in to change notification settings - Fork 25.6k
Description
Elasticsearch version (bin/elasticsearch --version
):
Version: 7.3.0, Build: default/rpm/de777fa/2019-07-24T18:30:11.767338Z, JVM: 12.0.1
Plugins installed: []
repository-s3
JVM version (java -version
):
JVM shipped with Elasticsearch (rpm)
OS version (uname -a
if on a Unix-like system):
Linux es-hot-03.aegaeon-it.com 4.9.120-xxxx-std-ipv6-64 #327490 SMP Thu Aug 16 10:11:35 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Description of the problem including expected versus actual behavior:
We noticed a severe degradation of date_histogram
performance after upgrading from Elasticsearch 6.8.2
to the latest 7.3.0
.
The table below shows the execution time of the same query on the same cluster before and after the upgrade. The first execution happens after caches are cleared as follows:
- ES caches are cleared with
POST _cache/clear
- Filesystem caches are invalidated by doing a cat of all files found in
/var/lib/elasticsearch
do/dev/null
(that's the best we found...)
Timings are as follows:
ES Version Timings (ms)
------------------------------------
6.8.2 10388, 5797, 7319
7.3.0 21553, 15138, 14833
As you notice 7.3.0
is at least twice slower than 6.8.2
.
The dataset is made of about 650m
documents spread evenly between 15 indexes. Indexes have 3 shards each and no replica. The cluster is made of three nodes, each with 7Gb RAM (of each 4Gb is allocated to the heap) and 2 vCPUs at 3.1Ghz. There is no activity on the cluster besides this test query.
The query targets only half of the documents (using a date_range) and builds a date_histogram
with buckets of 3h
. This query is actually what KIbana's Discover panel will do...
{
"size": 500,
"sort": [
{
"@timestamp": {
"order": "desc",
"unmapped_type": "boolean"
}
}
],
"aggs": {
"2": {
"date_histogram": {
"field": "@timestamp",
"interval": "3h",
"time_zone": "Europe/Berlin",
"min_doc_count": 1
}
}
},
"docvalue_fields": [
{
"field": "@timestamp",
"format": "date_time"
}
],
"query": {
"bool": {
"must": [
{
"range": {
"@timestamp": {
"format": "strict_date_optional_time",
"gte": "2019-01-09T00:00:00.000Z",
"lte": "2019-01-15T00:00:00.000Z"
}
}
}
]
}
}
}
We are very surprised by this drop of performance...
Did we forgot to change some configurations parameters when doing the upgrade or is it a regression in ES itself ?