Add proposal for parallel compaction by time interval #4272
Conversation
roystchiang
force-pushed
the
compactor-proposal
branch
from
d94b281 to
b386d7b
Compare
There was a problem hiding this comment.
I didn't follow this. What sort of corruption are you thinking about?
Is it related to this? thanos-io/thanos#4046
There was a problem hiding this comment.
I'm thinking of #3569
I suppose it's not exactly a corruption, but something that prevents compactor from compacting currently.
There was a problem hiding this comment.
I'm not sure I fully understand the proposed approach here. You can still compact all non-corrupted blocks together and just leave out of the compaction the corrupted block.
There was a problem hiding this comment.
I did not want to introduce a different behavior from what is currently happening. I wanted the proposal to be scoped to introducing horizontal scaling, and we can work on skipping bad blocks at a later stage.
There was a problem hiding this comment.
That idea seems fine. The amount of detail in the example seems overkill to make the point.
There was a problem hiding this comment.
That Thanos PR seems to be stalled; looks like Bartek called for a redesign.
The Google doc is just laying out the goals; it just says "TBD" for what would actually be done.
There was a problem hiding this comment.
Do you think it still makes sense to reference it given that it is not really actively being developed?
There was a problem hiding this comment.
Definitely, always good to see where others have been before.
I guess I mostly wanted to clarify that this isn't something we can build on directly.
There was a problem hiding this comment.
I'm not sure I fully understand the proposed approach here. You can still compact all non-corrupted blocks together and just leave out of the compaction the corrupted block.
Signed-off-by: Roy Chiang <[email protected]>
Signed-off-by: Roy Chiang <[email protected]>
roystchiang
force-pushed
the
compactor-proposal
branch
from
13855cd to
b04d90f
Compare
There was a problem hiding this comment.
That idea seems fine. The amount of detail in the example seems overkill to make the point.
There was a problem hiding this comment.
Definitely, always good to see where others have been before.
I guess I mostly wanted to clarify that this isn't something we can build on directly.
| --- | ||
|
|
||
| ## Introduction | ||
| As a part of pushing Cortex’s scaling capability at AWS, we have done performance testing with Cortex and found the compactor to be one of the main limiting factors for higher active timeseries limit per tenant. The documentation [Compactor](https://cortexmetrics.io/docs/blocks-storage/compactor/#how-compaction-works) describes the responsibilities of a compactor, and this proposal focuses on the limitations of the current compactor architecture. In the current architecture, compactor has simple sharding, meaning that a single tenant is sharded to a single compactor. In addition, a compactor handles compaction groups of a single tenant iteratively, meaning that blocks belonging non-overlapping times are not compacted in parallel. |
There was a problem hiding this comment.
"compaction groups" is not defined before use, either in this document or at the referenced link.
Possibly it could go in https://github.com/cortexproject/cortex/blob/master/docs/guides/glossary.md
| As a part of pushing Cortex’s scaling capability at AWS, we have done performance testing with Cortex and found the compactor to be one of the main limiting factors for higher active timeseries limit per tenant. The documentation [Compactor](https://cortexmetrics.io/docs/blocks-storage/compactor/#how-compaction-works) describes the responsibilities of a compactor, and this proposal focuses on the limitations of the current compactor architecture. In the current architecture, compactor has simple sharding, meaning that a single tenant is sharded to a single compactor. In addition, a compactor handles compaction groups of a single tenant iteratively, meaning that blocks belonging non-overlapping times are not compacted in parallel. | ||
|
|
||
| ### Problem and Requirements | ||
| Currently, a compactor is able to compact up to 20M timeseries within 2 hours for a level-2 compaction, including the time to download blocks, compact, and upload the newly compacted block. We would like to increase the timeseries limit per tenant, and compaction is one of the limiting factors. In addition, we would like to achieve the following: |
There was a problem hiding this comment.
I cannot find a definition of "level-2" here or in Compactor docs either.
|
I'm going to merge the proposal to move forward but I would be glad if you could address Bryan comments in a follow up PR. Thanks! |
|
@roystchiang gentle reminder I made some points about explaining terms, and I couldn't see any follow-up. |
I will submit a follow up PR this week. Thanks for the reminder. |
3 participants
What this PR does:
Add proposal for parallel compaction by time interval
Which issue(s) this PR fixes:
Proposal to work on a fix for #3753
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]