Description
Describe the bug
GetSeries and GetLabels API can retrieve informations from Ingesters or/and storegateways (when query_store_for_labels_enabled
configuration is enabled).
We have different behaviors if the requests are going to Ingesters
and to storegateways
.
On the first case, the max_fetched_series_per_query
is not enforced at all while in latter case this configuration is enforced:
StoreGateway:
cortex/pkg/querier/blocks_store_queryable.go
Line 625 in 45f043f
Ingester:
cortex/pkg/distributor/distributor.go
Line 958 in 45f043f
This behavior can cause OOM on the queriers as there is a unbounded limit for number of series brought from ingesters.
To Reproduce
Steps to reproduce the behavior:
- Start Cortex (SHA or version) : any
- Configure
query_store_for_labels_enabled
to 10 - Send more than 10 series
- call
/series
withmatch[]={__name__=~".+"}
Expected behavior
max_fetched_series_per_query
limit should be respected on both cases (data coming from StoreGateway or Ingester)
Storage Engine
- Blocks
- Chunks
Additional Context