Query Partial Data

[justinjung04]

What this PR does:

This PR introduces new limits (tenant config) of query_partial_data and rules_partial_data, which allows tenants to receive 2xx with a warning message when querying ingesters fail in zones greater than max unavailable zones, but less than all available zones.

For example, in the following scenario,

• Total zone count: 3
• Max unavailable zone: 1
• Zones where ingesters are failing: 2

Current behavior always returns 5xx:

But with this feature enabled, customers will receive data with warning message "Query result may contain partial data".

This is useful for cortex setup that replicates ingested data across different zones. In this case, even if ingesters from different zones go down, there is a high probability of full data available in ingesters from the remaining zone(s). In such case, tenants would rather want to receive 2xx than 5xx.

To summarize:

1. Ingesters failing across 0 to MaxUnavailableZones will return status code 200
2. Ingesters failing across MaxUnavailableZones to TotalZones - 1 will return status code 200 with a warning message
3. Ingesters failing in all zones will return status code 500

Which issue(s) this PR fixes:
n/a

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[oberkem]

I'm assuming this defaults to getting response from at least 1 ingester.

should this also be configurable and always needs to be lower than replication_factor? would a user want 2 out of 5 replica instead of 1 out of 5?

I think at least 1 is a good start. if the need emerges, we can re-consider

[justinjung04]

I agree. The base assumption of this feature is that zonal replication is reliable, so I wanted to keep the config nice and simple. We can definitely re-iterate in the future if needed.

[CharlieTLe]

Hello @justinjung04, thank you for opening this PR.

There is a release in progress. As such, please rebase your CHANGELOG entry on top of the master branch and move the CHANGELOG entry to the top under ## master / unreleased.

Thanks,
Charlie

[justinjung04]

Thank you @CharlieTLe, I've rebased the change.

[harry671003]

Great work 👍

[alanprot]

Amazing!

LGTM! thanks for doing this!

[yeya24]

Thanks. Looks good.
The only thing is that we might want to also apply partial data response to metric metadata and query exemplars call