-
Notifications
You must be signed in to change notification settings - Fork 832
Closed
Description
Describe the bug
https://github.com/cortexproject/cortex/actions/runs/13108526988/job/36567335809?pr=6571
There is a data race detected in the test.
Write:
https://github.com/cortexproject/cortex/blob/master/pkg/ingester/user_state.go#L224
Read:
https://github.com/cortexproject/cortex/blob/master/pkg/ingester/user_state.go#L138
==================
WARNING: DATA RACE
Write at 0x00c00aafeec0 by goroutine 16642:
github.com/cortexproject/cortex/pkg/ingester.(*labelSetCounter).increaseSeriesLabelSet()
/__w/cortex/cortex/pkg/ingester/user_state.go:224 +0x273
github.com/cortexproject/cortex/pkg/ingester.(*userTSDB).PostCreation()
/__w/cortex/cortex/pkg/ingester/ingester.go:472 +0x1a7
github.com/prometheus/prometheus/tsdb.(*stripeSeries).getOrSet()
/__w/cortex/cortex/vendor/github.com/prometheus/prometheus/tsdb/head.go:2031 +0x2cd
github.com/prometheus/prometheus/tsdb.(*Head).getOrCreateWithID()
/__w/cortex/cortex/vendor/github.com/prometheus/prometheus/tsdb/head.go:1703 +0x13d
github.com/prometheus/prometheus/tsdb.(*Head).getOrCreate()
/__w/cortex/cortex/vendor/github.com/prometheus/prometheus/tsdb/head.go:1699 +0xe4
github.com/prometheus/prometheus/tsdb.(*headAppender).getOrCreate()
/__w/cortex/cortex/vendor/github.com/prometheus/prometheus/tsdb/head_append.go:438 +0x4ab
github.com/prometheus/prometheus/tsdb.(*headAppender).Append()
/__w/cortex/cortex/vendor/github.com/prometheus/prometheus/tsdb/head_append.go:334 +0x13b
github.com/prometheus/prometheus/tsdb.(*initAppender).Append()
/__w/cortex/cortex/vendor/github.com/prometheus/prometheus/tsdb/head_append.go:52 +0x137
github.com/prometheus/prometheus/tsdb.(*dbAppender).Append()
<autogenerated>:1 +0xa1
github.com/cortexproject/cortex/pkg/ingester.(*Ingester).Push()
/__w/cortex/cortex/pkg/ingester/ingester.go:1305 +0x3a82
github.com/cortexproject/cortex/pkg/ingester.TestPushRace.func2()
/__w/cortex/cortex/pkg/ingester/ingester_test.go:707 +0x364
Previous read at 0x00c00aafeec0 by goroutine 16613:
github.com/cortexproject/cortex/pkg/ingester.(*labelSetCounter).backFillLimit()
/__w/cortex/cortex/pkg/ingester/user_state.go:138 +0x667
github.com/cortexproject/cortex/pkg/ingester.(*userTSDB).PreCreation.(*labelSetCounter).canAddSeriesForLabelSet.func1()
/__w/cortex/cortex/pkg/ingester/user_state.go:128 +0x39e
github.com/cortexproject/cortex/pkg/ingester.(*Limiter).AssertMaxSeriesPerLabelSet()
/__w/cortex/cortex/pkg/ingester/limiter.go:118 +0x2a4
github.com/cortexproject/cortex/pkg/ingester.(*labelSetCounter).canAddSeriesForLabelSet()
/__w/cortex/cortex/pkg/ingester/user_state.go:118 +0x548
github.com/cortexproject/cortex/pkg/ingester.(*userTSDB).PreCreation()
/__w/cortex/cortex/pkg/ingester/ingester.go:451 +0x3e9
github.com/prometheus/prometheus/tsdb.(*stripeSeries).getOrSet()
/__w/cortex/cortex/vendor/github.com/prometheus/prometheus/tsdb/head.go:2004 +0xa2
github.com/prometheus/prometheus/tsdb.(*Head).getOrCreateWithID()
/__w/cortex/cortex/vendor/github.com/prometheus/prometheus/tsdb/head.go:1703 +0x13d
github.com/prometheus/prometheus/tsdb.(*Head).getOrCreate()
/__w/cortex/cortex/vendor/github.com/prometheus/prometheus/tsdb/head.go:1699 +0xe4
github.com/prometheus/prometheus/tsdb.(*headAppender).getOrCreate()
/__w/cortex/cortex/vendor/github.com/prometheus/prometheus/tsdb/head_append.go:438 +0x4ab
github.com/prometheus/prometheus/tsdb.(*headAppender).Append()
/__w/cortex/cortex/vendor/github.com/prometheus/prometheus/tsdb/head_append.go:334 +0x13b
github.com/prometheus/prometheus/tsdb.(*dbAppender).Append()
<autogenerated>:1 +0xa1
github.com/cortexproject/cortex/pkg/ingester.(*Ingester).Push()
/__w/cortex/cortex/pkg/ingester/ingester.go:1305 +0x3a82
github.com/cortexproject/cortex/pkg/ingester.TestPushRace.func2()
/__w/cortex/cortex/pkg/ingester/ingester_test.go:707 +0x364
Goroutine 16642 (running) created at:
github.com/cortexproject/cortex/pkg/ingester.TestPushRace()
/__w/cortex/cortex/pkg/ingester/ingester_test.go:705 +0x1224
testing.tRunner()
/usr/local/go/src/testing/testing.go:1690 +0x226
testing.(*T).Run.gowrap1()
/usr/local/go/src/testing/testing.go:1743 +0x44
Goroutine 16613 (running) created at:
github.com/cortexproject/cortex/pkg/ingester.TestPushRace()
/__w/cortex/cortex/pkg/ingester/ingester_test.go:705 +0x1224
testing.tRunner()
/usr/local/go/src/testing/testing.go:1690 +0x226
testing.(*T).Run.gowrap1()
/usr/local/go/src/testing/testing.go:1743 +0x44
==================
--- FAIL: TestPushRace (3.77s)
testing.go:1399: race detected during execution of test
To Reproduce
Steps to reproduce the behavior:
- Start Cortex (SHA or version)
- Perform Operations(Read/Write/Others)
Expected behavior
A clear and concise description of what you expected to happen.
Environment:
- Infrastructure: [e.g., Kubernetes, bare-metal, laptop]
- Deployment tool: [e.g., helm, jsonnet]
Additional Context