Set a max concurrency TSDB blocks compaction

[pracucci]

Every two hours (by default) each ingester cuts blocks from head nearly at the same time. Each ingester does it for any tenant, because we have 1 TSDB opened for each tenant. We're observing an increased latency in the ingesters when this occur (other than an increased memory usage): the more tenants are writing to an ingester, the more this problem is apparent.

We could call DisableCompactions() on each TSDB right after creation and have a loop inside Cortex to periodically call Compact() on each TSDB (Compact() has been made public in the PR prometheus/prometheus#6667). This way we can control the max concurrency.

[pstibrany]

I have a code ready at https://github.com/pstibrany/cortex/tree/tsdb-blocks-compaction-concurrency, but waiting for PR #1970 to be merged first, as tests currently don't pass due to Prometheus vs Thanos mismatch.

[pracucci]

but waiting for PR #1970 to be merged first

@pstibrany The PR #1970 has been merged.

[pstibrany]

Waiting for #2088 now.

[brancz]

I can't tell at a glance from your implementation, but wanted to share my thoughts: we're thinking about this on Thanos receive as well, and we're thinking about choosing a random, well-distributed value within the min block size for any TSDB instance, that way not only within one process handling multiple TSDBs has a well-distribution but also across processes.

[pstibrany]

@brancz My implementation is currently very simple... idea is to disable automatic compactions, because we don't have any control over those, and simply run compaction once every defined interval on all open TSDBs (we have one per tenant). This way we can run only limited number of compactions at once.

This is going to change a bit, to tie it to the shipping process – to compact and then ship. Right now these two are separate in my implementation.

Just to be clear, this is not about compacting stored blocks, but HEAD compactions only.

[brancz]

Just to be clear, this is not about compacting stored blocks, but HEAD compactions only.

Yes we're thinking about exactly the same.

[pracucci]

we're thinking about this on Thanos receive as well, and we're thinking about choosing a random, well-distributed value within the min block size for any TSDB instance, that way not only within one process handling multiple TSDBs has a well-distribution but also across processes.

What's the goal of spreading Head compactions across the time across different receiver replicas?

The use case we have on Cortex is different. In a multi-tenant system we end up compacting Head for multiple TSDB on the same ingester instance at the same time. We want to avoid 1K TSDB all compacting from head at the same time on the same node.

[brancz]

It's to spread the pressure on the network link(s), so that uploads which have very high bandwidth characteristics don't burst all at once when all head blocks are cut more or less simultaneously by all receivers.

[pracucci]

It's to spread the pressure on the network link(s), so that uploads which have very high bandwidth characteristics don't burst all at once when all head blocks are cut more or less simultaneously by all receivers.

I would then argue that what you want is to coordinate the shipper, not head compactions. Am I wrong?

[brancz]

Yes you have a point, we've been looking at the block cutting and upload as one unit, but you're right that's indeed the part that I'm talking about.

[pracucci]

Yes you have a point, we've been looking at the block cutting and upload as one unit, but you're right that's indeed the part that I'm talking about.

In Cortex we've de-coupled them. This issue is about the head block cutting, while for the shipping what we've done so far is introducing a per-ingester max concurrency limit. There's no coordination across different ingesters, but in each ingester we've limited the concurrency (it's a bit different than your use case, because we run 1 shipper for each tenant, so it's probably an issue you don't have).

I would be very interested into learn if/how you will solve the coordination problem. Generally speaking, we try to avoid any real coordination for this kind of problems, and we usually adopt random jitters to approximately distribute the load.