resource based throttling: reject only adhoc queries

[erlan-z]

What this PR does:
Exclude rules queries from resource-based query rejection

Description:
This change improves the resource-based query rejection mechanism by ensuring that only ad-hoc API queries are rejected when resource thresholds are breached. Queries originating from rule evaluations are now excluded from rejection.

Introduced RequestSource metadata to distinguish between API and ruler queries.

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[justinjung04]

Thank you!

[justinjung04]

nit: what do you think of adding helper functions in requestmeta itself? ie. requestmeta.RequestFromApi(ctx) and requestmeta.RequestFromRuler(ctx)

[justinjung04]

LGTM, thank you

[SungJin1212]

How about reusing tripperware.SourceAPI and tripperware.SourceRuler?

[yeya24]

As a feature that intends to protect storage layer, I am not sure if it is the best way to hardcode it by differentiating source of requests and only rejecting one type but not others. I understand the use case behind this change. But if we go with this, it opens a door that we may need to extend and support throttling based on other types of source such as user agent, dashboard ID, etc.

I think a better approach is to go with #6922 and throttle heavy queries no matter its source.

[erlan-z]

As a feature that intends to protect storage layer, I am not sure if it is the best way to hardcode it by differentiating source of requests and only rejecting one type but not others. I understand the use case behind this change. But if we go with this, it opens a door that we may need to extend and support throttling based on other types of source such as user agent, dashboard ID, etc.

I think a better approach is to go with #6922 and throttle heavy queries no matter its source.

Thanks for the review. I agree #6922 is the right direction to avoid hard-coding on user attributes like User-Agent or dashboard ID. That said, I see ruler vs. ad-hoc queries as a different category: it’s a system-level distinction, whereas the others are user-driven. So I don’t think this opens the door to extending based on those other characteristics.

I also recognize that ruler queries can sometimes be heavy, but that’s a rare corner case and not really the problem this feature is trying to solve. Because #6922 is more of a best-effort mechanism, I’d even suggest carrying this system-level distinction into that work as well, to avoid ruler queries being rejected in cases where precision isn’t perfect.

What do you think?

[yeya24]

That said, I see ruler vs. ad-hoc queries as a different category: it’s a system-level distinction, whereas the others are user-driven

I don't think we can easily identify queries coming from API as ad-hoc queries. There are different usecases that might be even more important than rules and it is all up to the user to decide the priority. Even rules can come from the query API endpoint from some remote evaluators like Thanos Ruler.

[erlan-z]

That said, I see ruler vs. ad-hoc queries as a different category: it’s a system-level distinction, whereas the others are user-driven

I don't think we can easily identify queries coming from API as ad-hoc queries. There are different usecases that might be even more important than rules and it is all up to the user to decide the priority. Even rules can come from the query API endpoint from some remote evaluators like Thanos Ruler.

Makes sense, thanks. I’m aligned with #6922. I realize it’s not always clear which queries are higher priority, and there can certainly be important API queries too, but since we can’t reliably identify those, it might help to at least avoid rejecting ruler queries, which are usually the priority ones we can identify in practice. Happy to leave this out here and revisit if needed.