-
Notifications
You must be signed in to change notification settings - Fork 818
Ignore the queries in the future in the ingesters #1929
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ignore the queries in the future in the ingesters #1929
Conversation
This is a temp workaround until cortexproject#1929 is merged and deployed. Signed-off-by: Goutham Veeramachaneni <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Few questions, please:
- Why don't we enforce it in the querier, instead of duplicating the logic between the chunk store and ingester?
- Shouldn't we apply the same fix to ingester's
Query()
andv2Query()
? - Can we add unit tests, to avoid regressions?
I'm talking specifically about |
My sense is that the chunk store does it to avoid generating error messages about "missing tables" when the problem is in the query parameters. |
What about wrapping the distributor queryable in order to short circuit these before they call the ingesters? Maybe here: |
866df34
to
414224f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
code lgtm, but I can't really comment on whether this is the right fix for what we were seeing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM with few nits.
95a7926
to
24f1ea2
Compare
24f1ea2
to
ee9b67b
Compare
Lots of changes since. Owen's PR got merged.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! (just left a couple of nits)
CHANGELOG.md
Outdated
* [CHANGE] We now ignore queries that are more than 10mins into the future. #1929 | ||
* Can be configured using `querier.max-query-into-future`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm wondering if it would be more clear:
* [CHANGE] We now ignore queries that are more than 10mins into the future. #1929 | |
* Can be configured using `querier.max-query-into-future`. | |
* [CHANGE] We now enforce queries to be up to `-querier.max-query-into-future` into the future (defaults to 10m). #1929 |
Because we don't "ignore queries" but we actually clamp the max timestamp.
pkg/querier/querier_test.go
Outdated
maxQueryIntoFuture: 10 * time.Minute, | ||
}, | ||
{ // Skipping stores is disabled. | ||
name: "hit-test2", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Clashing name with the previous one. The name may be something like "max-query-into-future-disabled".
Signed-off-by: Goutham Veeramachaneni <[email protected]>
ee9b67b
to
b784a85
Compare
We ignore queries in the future now.
Signed-off-by: Goutham Veeramachaneni [email protected]
What this PR does:
Which issue(s) this PR fixes:
Fixes #
Checklist
CHANGELOG.md
updated - the order of entries should be[CHANGE]
,[FEATURE]
,[ENHANCEMENT]
,[BUGFIX]