func (i *Ingester) Push has some performance issues

[t00350320]

Describe the bug
1、db.activeSeries.UpdateSeries take a second hash operation for one series label in fingerprint(series).
fingerprint(series)
while the first hash is in
ref, copiedLabels := app.GetRef(cortexpb.FromLabelAdaptersToLabels(ts.Labels))
func (a *headAppender) GetRef(lset labels.Labels) (storage.SeriesRef, labels.Labels) {
s := a.head.series.getByHash(lset.Hash(), lset)
In 15M series scenario, two times hash take lots time in Flame Graph like this:

2、sync.Pool is used in
fingerprint(series)
while it's on the contrary in
s := a.head.series.getByHash(lset.Hash(), lset)

3、db.activeSeries.UpdateSeries takes a hard code "stripe size"
const ( numActiveSeriesStripes = 512 )
so, a dynamic configuration like
StripeSize: i.cfg.BlocksStorageConfig.TSDB.StripeSize,
may speed up the active series labels.Equal in
findEntryForSeries

To Reproduce
Steps to reproduce the behavior:

1. Start Cortex (SHA or version)
2. Perform Operations(Write/Others)

16C128G*3, tartget=all,
write ops : 15M series / 30s

[songjiayang]

good watch point

[t00350320]

For issue 1 and 2, we tryed to calculate the labelhash in advance，and then add new function like 'app.GetRefByHash' in vendor/prometheus , the total CPU cost decrease about 5% in our env. More details will be posted here...

[t00350320]

benchmark datas for BenchmarkIngesterPush.
(this benchmark test case may disable i.cfg.ActiveSeriesMetricsEnabled, so you'll enable it manually )

1. current version

BenchmarkIngesterPush-12
2 507131850 ns/op 38567308 B/op 70120 allocs/op
PASS
ok github.com/cortexproject/cortex/pkg/ingester 2.626s

BenchmarkIngesterPush-12
5 232420280 ns/op 16058694 B/op 28116 allocs/op
PASS
ok github.com/cortexproject/cortex/pkg/ingester 4.317s

BenchmarkIngesterPush-12
5 228980180 ns/op 15501105 B/op 28094 allocs/op
PASS
ok github.com/cortexproject/cortex/pkg/ingester 4.674s

BenchmarkIngesterPush-12
3 347628100 ns/op 25832381 B/op 46776 allocs/op
PASS
ok github.com/cortexproject/cortex/pkg/ingester 2.573s

function costs 507131850,232420280,228980180,347628100

2、compute hash  in advance , then use it in

ref, copiedLabels := app.GetRefByhash(tslabels, tslabhash)
and
db.activeSeries.UpdateSeriesByHash

BenchmarkIngesterPush-12
6 215605800 ns/op 12859718 B/op 23423 allocs/op
PASS
ok github.com/cortexproject/cortex/pkg/ingester 3.770s

BenchmarkIngesterPush-12
5 212980600 ns/op 15503232 B/op 28101 allocs/op
PASS
ok github.com/cortexproject/cortex/pkg/ingester 3.666s

BenchmarkIngesterPush-12
6 187246967 ns/op 12858720 B/op 23421 allocs/op
PASS
ok github.com/cortexproject/cortex/pkg/ingester 4.601s

BenchmarkIngesterPush-12
5 212203840 ns/op 15791737 B/op 28099 allocs/op
PASS
ok github.com/cortexproject/cortex/pkg/ingester 3.586s

function costs 215605800,212980600,187246967,212203840

So, it seems the modified version reduce CPU costs. We also demostrate it in pprof Flame Graph because there is only one Label.Hash compared with previous issue report in part1.

[t00350320]

Please @alanprot @yeya24 help to review

[bboreham]

Tip: your benchmark results will be easier to read if you use benchstat.

[t00350320]

Tip: your benchmark results will be easier to read if you use benchstat.

thanks, benchstat results:

[t00350320]

BenchmarkActiveSeries_UpdateSeries benchstat:

[yeya24]

I will close this one as the code was merged. Thanks!