Expose Prometheus TSDB compaction metrics

[pracucci]

The following TSDB metrics are not exposed by Cortex (blocks storage) but are actually quite important to be able to graph at which point in time TSDB head compactions occur:

prometheus_tsdb_compactions_triggered_total
prometheus_tsdb_compactions_failed_total

We should add them to the mapping in pkg/ingester/metrics.go and update unit tests accordingly.

[pstibrany]

I planned to include this in #2172, but forgot. As I'm going to do some changes in that PR, I will include it.

[pstibrany]

Do you think that it would be interesting to include prometheus_tsdb_compactions_skipped_total ("Total number of skipped compactions due to disabled auto compaction."), since we are disabling auto compaction now? I'm thinking that even when we run compaction manually, this value shouldn't get very high, otherwise settings are not correct.

[pracucci]

Why the value of the skipped will not go high? If we try the compact every minute but the block range period is 2h, most of them will be skipped. Am I missing something?

…

On Mon, Feb 24, 2020, 09:46 Peter Štibraný ***@***.***> wrote: Do you think that it would be interesting to include prometheus_tsdb_compactions_skipped_total ("Total number of skipped compactions due to disabled auto compaction."), since we are disabling auto compaction now? I'm thinking that even when we run compaction manually, this value shouldn't get very high, otherwise settings are not correct. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2174?email_source=notifications&email_token=AAM7QEEUZOUXP7EPYKVXUA3REOCMXA5CNFSM4K2CCXFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMXAEQY#issuecomment-590217795>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAM7QEB264BXJRWSBA7GFJDREOCMXANCNFSM4K2CCXFA> .

[pstibrany]

Why the value of the skipped will not go high? If we try the compact every minute but the block range period is 2h, most of them will be skipped. Am I missing something?

OK, looking at the code, I think misunderstood head compaction – TSDB code indeed looks like it tries to do it every minute (!?), or on appender's commit under certain conditions – so it would get high pretty quickly.

[pstibrany]

prometheus_tsdb_compactions_triggered_total
prometheus_tsdb_compactions_failed_total

Turns out these are actually not very interesting. TSDB tries to "trigger" compaction all the time, but since autocompactions are disabled, nothing happens.

I think we should count these internally in Cortex, and increase them when we actually run compaction, and on failures.

[pstibrany]

I think we should count these internally in Cortex, and increase them when we actually run compaction, and on failures.

This is what I did in #2172.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[pstibrany]

This was fixed by #2172

[pstibrany]

cortex_ingester_tsdb_compactions_triggered_total
cortex_ingester_tsdb_compactions_failed_total