address comments

Author: justinjung04

CHANGELOG.md

@@ -5,7 +5,6 @@
 * [FEATURE] Query Frontend: Add dynamic interval size for query splitting. This is enabled by configuring experimental flags `querier.max-shards-per-query` and/or `querier.max-fetched-data-duration-per-query`. The split interval size is dynamically increased to maintain a number of shards and total duration fetched below the configured values. #6458
 * [FEATURE] Querier/Ruler: Add `query_partial_data` and `rules_partial_data` limits to allow queries/rules to be evaluated with data from a single zone, if other zones are not available. #6526
 * [FEATURE] Update prometheus alertmanager version to v0.28.0 and add new integration msteamsv2, jira, and rocketchat. #6590
-* [FEATURE] Ingester: Add a `-ingester.enable-ooo-native-histograms` flag to enable out-of-order native histogram ingestion per tenant. It only takes effect when `-blocks-storage.tsdb.enable-native-histograms=true` and `-ingester.out-of-order-time-window` > 0. It is applied after the restart if it is changed at runtime through the runtime config. #6626
 * [FEATURE] Ingester/StoreGateway: Add `resource-thresholds` in ingesters and store gateways to throttle query requests when the pods are under resource pressure. #6674
 * [FEATURE] Ingester: Support out-of-order native histogram ingestion. It automatically enabled when `-ingester.out-of-order-time-window > 0` and `-blocks-storage.tsdb.enable-native-histograms=true`. #6626 #6663
 * [ENHANCEMENT] Alertmanager: Add nflog and silences maintenance metrics. #6659

docs/configuration/config-file-reference.md

@@ -273,10 +273,14 @@ query_scheduler:
 
 resource_thresholds:
   # Utilization threshold for CPU in percentage, between 0 and 1. 0 to disable.
+  # The CPU utilization metric is from github.com/prometheus/procfs, which is a
+  # close estimate. Applicable to ingesters and store-gateways only.
   # CLI flag: -resource-thresholds.cpu
   [cpu: <float> | default = 0]
 
   # Utilization threshold for heap in percentage, between 0 and 1. 0 to disable.
+  # The heap utilization metric is from runtime/metrics, which is a close
+  # estimate. Applicable to ingesters and store-gateways only.
   # CLI flag: -resource-thresholds.heap
   [heap: <float> | default = 0]
 

docs/guides/resource-based-throttling.md

@@ -0,0 +1,55 @@
+---
+title: "Protecting Cortex from Heavy Queries"
+linkTitle: "Protecting Cortex from Heavy Queries"
+weight: 11
+slug: protecting-cortex-from-heavy-queries
+---
+
+PromQL is powerful, and is able to result in query requests that have very wide range of data fetched and samples processed. Heavy queries can cause:
+
+1. CPU on any query component to be partially exhausted, increasing latency and causing incoming queries to queue up with high chance of time-out.
+2. CPU on any query component to be fully exhausted, causing GC to slow down leading to the pod being out-of-memory and killed.
+3. Heap memory on any query component to be exhausted, leading to the pod being out-of-memory and killed.
+
+It's important to protect Cortex components by setting appropriate limits and throttling configurations based on your infrastructure and data ingested by the customers.
+
+## Static limits
+
+There are number of static limits that you could configure to block heavy queries from running.
+
+### Max outstanding requests per tenant
+
+See https://cortexmetrics.io/docs/configuration/configuration-file/#query_frontend_config:~:text=max_outstanding_requests_per_tenant for details.
+
+### Max data bytes fetched per (sharded) query
+
+See https://cortexmetrics.io/docs/configuration/configuration-file/#query_frontend_config:~:text=max_fetched_data_bytes_per_query for details.
+
+### Max series fetched per (sharded) query
+
+See https://cortexmetrics.io/docs/configuration/configuration-file/#query_frontend_config:~:text=max_fetched_series_per_query for details.
+
+### Max chunks fetched per (sharded) query
+
+See https://cortexmetrics.io/docs/configuration/configuration-file/#query_frontend_config:~:text=max_fetched_chunk_bytes_per_query for details.
+
+### Max samples fetched per (sharded) query
+
+See https://cortexmetrics.io/docs/configuration/configuration-file/#querier_config:~:text=max_samples for details.
+
+## Resource-based throttling
+
+Although the static limits are able to protect Cortex components from specific query patterns, they are not generic enough to cover different combinations of bad query patterns. For example, what if the query fetches relatively large postings, series and chunks that are slightly below the individual limits? For a more generic solution, you can enable resource-based throttling by setting CPU and heap utilization thresholds.
+
+Currently, it only throttles incoming query requests with error code 429 (too many requests) when the resource usage breaches the configured thresholds.
+
+For example, the following configuration will start throttling query requests if either CPU or heap utilization is above 80%, leaving 20% of room for inflight requests.
+
+```
+target: ingester
+resource_thresholds:
+  cpu: 0.8
+  heap: 0.8
+```
+
+See https://cortexmetrics.io/docs/configuration/configuration-file/#generic-placeholders:~:text=resource_thresholds for details.

pkg/configs/resources.go

@@ -13,8 +13,8 @@ type Resources struct {
 }
 
 func (cfg *Resources) RegisterFlags(f *flag.FlagSet) {
-	f.Float64Var(&cfg.CPU, "resource-thresholds.cpu", 0, "Utilization threshold for CPU in percentage, between 0 and 1. 0 to disable.")
-	f.Float64Var(&cfg.Heap, "resource-thresholds.heap", 0, "Utilization threshold for heap in percentage, between 0 and 1. 0 to disable.")
+	f.Float64Var(&cfg.CPU, "resource-thresholds.cpu", 0, "Utilization threshold for CPU in percentage, between 0 and 1. 0 to disable. The CPU utilization metric is from github.com/prometheus/procfs, which is a close estimate. Applicable to ingesters and store-gateways only.")
+	f.Float64Var(&cfg.Heap, "resource-thresholds.heap", 0, "Utilization threshold for heap in percentage, between 0 and 1. 0 to disable. The heap utilization metric is from runtime/metrics, which is a close estimate. Applicable to ingesters and store-gateways only.")
 }
 
 func (cfg *Resources) Validate() error {

pkg/cortex/modules.go

@@ -777,6 +777,10 @@ func (t *Cortex) initResourceMonitor() (services.Service, error) {
 
 	scanner, err := resource.NewScanner()
 	if err != nil {
+		if errors.As(err, resource.UnsupportedOSError{}) {
+			level.Warn(util_log.Logger).Log("msg", "Skipping resource monitor", "err", err.Error())
+			return nil, nil
+		}
 		return nil, err
 	}
 

pkg/util/resource/monitor.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"fmt"
 	"net/http"
+	"runtime"
 	"runtime/metrics"
 	"sync"
 	"time"
@@ -24,6 +25,12 @@ func (e *ExhaustedError) Error() string {
 	return "resource exhausted"
 }
 
+type UnsupportedOSError struct{}
+
+func (e *UnsupportedOSError) Error() string {
+	return "resource scanner is only supported in linux"
+}
+
 const heapMetricName = "/memory/classes/Heap/objects:bytes"
 const monitorInterval = time.Second
 const dataPointsToAvg = 30
@@ -43,6 +50,10 @@ type Stats struct {
 }
 
 func NewScanner() (*Scanner, error) {
+	if runtime.GOOS != "linux" {
+		return nil, &UnsupportedOSError{}
+	}
+
 	proc, err := procfs.Self()
 	if err != nil {
 		return nil, errors.Wrap(err, "error reading proc directory")
@@ -89,8 +100,8 @@ type Monitor struct {
 
 	// Variables to calculate average CPU utilization
 	index         int
-	cpuRates      []float64
-	cpuIntervals  []float64
+	cpuRates      [dataPointsToAvg]float64
+	cpuIntervals  [dataPointsToAvg]float64
 	totalCPU      float64
 	totalInterval float64
 	lastCPU       float64
@@ -105,8 +116,8 @@ func NewMonitor(thresholds configs.Resources, limits configs.Resources, scanner
 		containerLimit: limits,
 		scanner:        scanner,
 
-		cpuRates:     make([]float64, dataPointsToAvg),
-		cpuIntervals: make([]float64, dataPointsToAvg),
+		cpuRates:     [dataPointsToAvg]float64{},
+		cpuIntervals: [dataPointsToAvg]float64{},
 
 		lock: sync.RWMutex{},
 	}