TSDB head compactor concurrency #2172

pstibrany · 2020-02-21T13:40:23Z

What this PR does: This PR makes TSDB head compaction interval and concurrency configurable (defaults to 2h). Shipping now happens right after compaction, and previous options to control shipping interval and concurrency have been removed. Shipping can still be disabled.

Which issue(s) this PR fixes:
Fixes #2003

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

gouthamve

Mostly LGTM, will 👍 after comments are addressed :)

CHANGELOG.md

pkg/storage/tsdb/config.go

gouthamve · 2020-02-24T10:45:28Z

docs/operations/blocks-storage.md

@@ -123,6 +123,14 @@ tsdb:
  # CLI flag: -experimental.tsdb.ship-concurrency
  [ship_concurrency: <int> | default = 10]

+  # How frequently does Cortex try to compact TSDB head. Block is only created if data covers smallest block range.
+  # CLI flag: -experimental.tsdb.head-compaction-interval
+  [head_compaction_interval: <duration> | default = 2h]


Hrm, this should be 5m and the tests should fail. Can you check why its not failing?

I have edited this file manually. Somehow it isn't covered by Marco's tool, but haven't looked into more detail why yet. I'll fix the default.

I think the reason is simply because block-storage is a separate file, while doc tool only generates config-file-reference.md. @pracucci surely knows better, I'm just guessing based on 5min looking around.

pracucci · 2020-02-24T13:15:56Z

Fixes #2174

pracucci

Good job @pstibrany and thanks for taking care of this! I left few comments, everything else LGTM 🎉

pracucci · 2020-02-24T13:18:49Z

pkg/ingester/ingester_v2.go

@@ -129,6 +145,11 @@ func NewV2(cfg Config, clientConfig client.Config, limits *validation.Overrides,
 	i.done.Add(1)
 	go i.updateLoop()

+	if i.cfg.TSDBConfig.HeadCompactionConcurrency > 0 && i.cfg.TSDBConfig.HeadCompactionInterval > 0 {


Why is it possible to disable compactions? We should never allow it. Even if you run without shipper you definitely wants compactions otherwise your memory will soon explode.

I would suggest:

Add a config validation to ensure that HeadCompactionConcurrency > 0

Always run compactionLoop()

What's your take?

I left an option to disable it when running tests. But now I think it doesn't make sense even in that scenario. I'll modify it.

pracucci · 2020-02-24T13:20:37Z

pkg/storage/tsdb/config.go

@@ -108,6 +112,8 @@ func (cfg *Config) RegisterFlags(f *flag.FlagSet) {
 	f.IntVar(&cfg.ShipConcurrency, "experimental.tsdb.ship-concurrency", 10, "Maximum number of tenants concurrently shipping blocks to the storage.")
 	f.StringVar(&cfg.Backend, "experimental.tsdb.backend", "s3", "TSDB storage backend to use")
 	f.IntVar(&cfg.MaxTSDBOpeningConcurrencyOnStartup, "experimental.tsdb.max-tsdb-opening-concurrency-on-startup", 10, "limit the number of concurrently opening TSDB's on startup")
+	f.DurationVar(&cfg.HeadCompactionInterval, "experimental.tsdb.head-compaction-interval", 5*time.Minute, "How frequently does Cortex try to compact TSDB head. Block is only created if data covers smallest block range.")


5m looks a bit unfrequent to me. This is a very inexpensive operation and I would suggest to run it more frequently: 30s or 1m.

Do we even need to make it configurable? I'd just hardcode 1m into the code.

I thought about hardcoding to 1m as well, and that was my initial comment. I then changed my mind because I think being able to configure it could help future integration tests (ie. compact every 5s to have integration tests running faster). What's your take?

P.S. I'm talking about integration tests cause I would like to add at least 1 integration test testing the compaction and shipping soon.

We need a secret option then ;-) I'd suggest adding a check to avoid setting it too high (> 5 mins).

We need a secret option then ;-)

We can have a secret YAML config option with doc:"hidden" but I'm not sure it's possible to have an hidden CLI flag (I haven't investigated).

pkg/storage/tsdb/config.go

Signed-off-by: Peter Štibraný <[email protected]>

This reverts commit ac372e9f065f00e909f67d7c3b3557e2974f311c. Signed-off-by: Peter Štibraný <[email protected]>

This reverts commit 9adb4cb9b4273ac189a3bb0478090966a0db67ec. Signed-off-by: Peter Štibraný <[email protected]>

Signed-off-by: Peter Štibraný <[email protected]>

To avoid confusion with compactor, which also has "compaction_interval" config field. Signed-off-by: Peter Štibraný <[email protected]>

Signed-off-by: Peter Štibraný <[email protected]>

They are now updated in Cortex code, when we call compaction. Signed-off-by: Peter Štibraný <[email protected]>

Signed-off-by: Peter Štibraný <[email protected]>

- decrease default to 1 min - compaction cannot be disabled - compaction interval must be greater than 0, and <= 5 mins Signed-off-by: Peter Štibraný <[email protected]>

pracucci

LGTM, with a nit. Thanks!

CHANGELOG.md

Signed-off-by: Peter Štibraný <[email protected]>

pstibrany mentioned this pull request Feb 24, 2020

Expose Prometheus TSDB compaction metrics #2174

Closed

gouthamve reviewed Feb 24, 2020

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

pkg/storage/tsdb/config.go Outdated Show resolved Hide resolved

gouthamve reviewed Feb 24, 2020

View reviewed changes

pracucci reviewed Feb 24, 2020

View reviewed changes

pstibrany added 19 commits February 24, 2020 15:09

Run local compactions on our own schedule and goroutines.

7b48087

Signed-off-by: Peter Štibraný <[email protected]>

Fix and add some tests to validation.

e5bd9fc

Signed-off-by: Peter Štibraný <[email protected]>

Refactor common code into runConcurrentUserWorkers method.

178f593

Signed-off-by: Peter Štibraný <[email protected]>

Tied compaction and shipping together.

7e9e382

Signed-off-by: Peter Štibraný <[email protected]>

Updated CHANGELOG.md

6440781

Signed-off-by: Peter Štibraný <[email protected]>

Updated CHANGELOG.md

1454ae1

Signed-off-by: Peter Štibraný <[email protected]>

Updated documentation.

d422afe

Signed-off-by: Peter Štibraný <[email protected]>

Merge blocks.

4fe63a1

Signed-off-by: Peter Štibraný <[email protected]>

Revert "Tied compaction and shipping together."

449f419

This reverts commit ac372e9f065f00e909f67d7c3b3557e2974f311c. Signed-off-by: Peter Štibraný <[email protected]>

Revert "Updated documentation."

0f60c01

This reverts commit 9adb4cb9b4273ac189a3bb0478090966a0db67ec. Signed-off-by: Peter Štibraný <[email protected]>

Fixed documentation.

6fee5b9

Signed-off-by: Peter Štibraný <[email protected]>

Renamed TSDB head compaction options to use "head" prefix.

341a653

To avoid confusion with compactor, which also has "compaction_interval" config field. Signed-off-by: Peter Štibraný <[email protected]>

Expose compaction metrics.

16ad22c

Signed-off-by: Peter Štibraný <[email protected]>

Moved compaction metrics to TSDB state.

853346b

They are now updated in Cortex code, when we call compaction. Signed-off-by: Peter Štibraný <[email protected]>

Decrease compaction interval.

286265c

Signed-off-by: Peter Štibraný <[email protected]>

Updated help and changelog entry.

4535ed3

Signed-off-by: Peter Štibraný <[email protected]>

Documentation

87cbc73

Signed-off-by: Peter Štibraný <[email protected]>

Fix default value.

8fee89b

Signed-off-by: Peter Štibraný <[email protected]>

Review feedback:

cd72ae9

- decrease default to 1 min - compaction cannot be disabled - compaction interval must be greater than 0, and <= 5 mins Signed-off-by: Peter Štibraný <[email protected]>

pracucci approved these changes Feb 24, 2020

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

pstibrany added 2 commits February 24, 2020 15:15

Fix CHANGELOG.md

c0de26d

Signed-off-by: Peter Štibraný <[email protected]>

Fix formatting to make lint happy.

10b65f9

Signed-off-by: Peter Štibraný <[email protected]>

gouthamve approved these changes Feb 25, 2020

View reviewed changes

pracucci merged commit 6d4da2c into cortexproject:master Feb 25, 2020

TSDB head compactor concurrency #2172

TSDB head compactor concurrency #2172

Uh oh!

Conversation

pstibrany commented Feb 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gouthamve left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pracucci commented Feb 24, 2020

Uh oh!

pracucci left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pracucci left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pstibrany commented Feb 21, 2020 •

edited

Loading