-
Notifications
You must be signed in to change notification settings - Fork 817
Cache Expended Posting on ingesters #6296
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
d33a2a8
to
9cdddd1
Compare
a5e3830
to
6937829
Compare
I love the test results! Thanks for doing this!! |
3 tasks
justinjung04
reviewed
Nov 3, 2024
Signed-off-by: alanprot <[email protected]>
Signed-off-by: alanprot <[email protected]>
Signed-off-by: alanprot <[email protected]>
Signed-off-by: alanprot <[email protected]>
Signed-off-by: alanprot <[email protected]>
6937829
to
34adf2a
Compare
…enabled Signed-off-by: alanprot <[email protected]>
34adf2a
to
d8d226d
Compare
alanprot
commented
Nov 4, 2024
yeya24
reviewed
Nov 4, 2024
d443d80
to
42e2235
Compare
Signed-off-by: alanprot <[email protected]>
42e2235
to
41aad71
Compare
…ready found keys. (cortexproject#6312) * Creating a test to show the race on the multilevel cache Signed-off-by: alanprot <[email protected]> * fix the race problem * Only fetch keys that were not found on the previous cache Signed-off-by: alanprot <[email protected]> --------- Signed-off-by: alanprot <[email protected]>
Signed-off-by: alanprot <[email protected]>
…tric Signed-off-by: alanprot <[email protected]>
yeya24
approved these changes
Nov 5, 2024
1 task
CharlieTLe
pushed a commit
to CharlieTLe/cortex
that referenced
this pull request
Dec 3, 2024
* Implementing Expanded Postings Cache Signed-off-by: alanprot <[email protected]> * small nit Signed-off-by: alanprot <[email protected]> * refactoring the cache so we dont need to call expire on every request Signed-off-by: alanprot <[email protected]> * Update total cache size when updating the item Signed-off-by: alanprot <[email protected]> * Fix fuzzy test after change the flag name Signed-off-by: alanprot <[email protected]> * remove max item config + create a new test case with only head cache enabled Signed-off-by: alanprot <[email protected]> * Documenting enabled as first field on the config Signed-off-by: alanprot <[email protected]> * Fix race on chunks multilevel cache + Optimize to avoid refetching already found keys. (cortexproject#6312) * Creating a test to show the race on the multilevel cache Signed-off-by: alanprot <[email protected]> * fix the race problem * Only fetch keys that were not found on the previous cache Signed-off-by: alanprot <[email protected]> --------- Signed-off-by: alanprot <[email protected]> * Improve Doc Signed-off-by: alanprot <[email protected]> * create new cortex_ingester_expanded_postings_non_cacheable_queries metric Signed-off-by: alanprot <[email protected]> --------- Signed-off-by: alanprot <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
ThisPR introduce caching expanded postings, similar to the approach used in SG, as seen in this change.
For such cases, a seed is added to the cache key, allowing invalidation whenever a new series is added or removed for a given metric name.
21|head|__name__=cpu_utilization|service=A
21|head|__name__=cpu_utilization|service=A|op=GET
On data ingestion, if a new series is added with
cpu_utilization
as the metric name, the seed changes, invalidating all cached entries for that metric.To control memory usage, a fixed-size seed slice is used. Metric names are hashed to determine their respective seed.
In load testing, this change demonstrated significant CPU and latency improvements on ingesters experiencing heavy query loads from rulers. In such cases, the same queries are evaluated constantly (every evaluation interval), and the postings are expanded for every iteration, leading to unnecessarily high CPU usage.
This are some graphs:

Flame Graphs Before and After (we can see that the after the CPU is being used mostly for the
Push
method, where before was mostly being used by theQueryStream
method:The idea was inspired on thanos-io/thanos#6420 and the promise based cache implementation by grafana/mimir-prometheus#14
Which issue(s) this PR fixes:
Fixes #
Checklist
CHANGELOG.md
updated - the order of entries should be[CHANGE]
,[FEATURE]
,[ENHANCEMENT]
,[BUGFIX]