Skip to content

max_fetched_series_per_query limits are not respected for /series API for recent data #4667

Closed
@alanprot

Description

@alanprot

Describe the bug
GetSeries and GetLabels API can retrieve informations from Ingesters or/and storegateways (when query_store_for_labels_enabled configuration is enabled).

We have different behaviors if the requests are going to Ingesters and to storegateways.
On the first case, the max_fetched_series_per_query is not enforced at all while in latter case this configuration is enforced:

StoreGateway:

limitErr := queryLimiter.AddSeries(cortexpb.FromLabelsToLabelAdapters(s.PromLabels()))

Ingester:

return client.MetricsForLabelMatchers(ctx, req)

This behavior can cause OOM on the queriers as there is a unbounded limit for number of series brought from ingesters.

To Reproduce
Steps to reproduce the behavior:

  1. Start Cortex (SHA or version) : any
  2. Configure query_store_for_labels_enabled to 10
  3. Send more than 10 series
  4. call /series with match[]={__name__=~".+"}

Expected behavior
max_fetched_series_per_query limit should be respected on both cases (data coming from StoreGateway or Ingester)

Storage Engine

  • Blocks
  • Chunks

Additional Context

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions