Skip to content

labelset data race #6572

@yeya24

Description

@yeya24

Describe the bug

https://github.com/cortexproject/cortex/actions/runs/13108526988/job/36567335809?pr=6571

There is a data race detected in the test.

Write:
https://github.com/cortexproject/cortex/blob/master/pkg/ingester/user_state.go#L224
Read:
https://github.com/cortexproject/cortex/blob/master/pkg/ingester/user_state.go#L138

==================
WARNING: DATA RACE
Write at 0x00c00aafeec0 by goroutine 16642:
  github.com/cortexproject/cortex/pkg/ingester.(*labelSetCounter).increaseSeriesLabelSet()
      /__w/cortex/cortex/pkg/ingester/user_state.go:224 +0x273
  github.com/cortexproject/cortex/pkg/ingester.(*userTSDB).PostCreation()
      /__w/cortex/cortex/pkg/ingester/ingester.go:472 +0x1a7
  github.com/prometheus/prometheus/tsdb.(*stripeSeries).getOrSet()
      /__w/cortex/cortex/vendor/github.com/prometheus/prometheus/tsdb/head.go:2031 +0x2cd
  github.com/prometheus/prometheus/tsdb.(*Head).getOrCreateWithID()
      /__w/cortex/cortex/vendor/github.com/prometheus/prometheus/tsdb/head.go:1703 +0x13d
  github.com/prometheus/prometheus/tsdb.(*Head).getOrCreate()
      /__w/cortex/cortex/vendor/github.com/prometheus/prometheus/tsdb/head.go:1699 +0xe4
  github.com/prometheus/prometheus/tsdb.(*headAppender).getOrCreate()
      /__w/cortex/cortex/vendor/github.com/prometheus/prometheus/tsdb/head_append.go:438 +0x4ab
  github.com/prometheus/prometheus/tsdb.(*headAppender).Append()
      /__w/cortex/cortex/vendor/github.com/prometheus/prometheus/tsdb/head_append.go:334 +0x13b
  github.com/prometheus/prometheus/tsdb.(*initAppender).Append()
      /__w/cortex/cortex/vendor/github.com/prometheus/prometheus/tsdb/head_append.go:52 +0x137
  github.com/prometheus/prometheus/tsdb.(*dbAppender).Append()
      <autogenerated>:1 +0xa1
  github.com/cortexproject/cortex/pkg/ingester.(*Ingester).Push()
      /__w/cortex/cortex/pkg/ingester/ingester.go:1305 +0x3a82
  github.com/cortexproject/cortex/pkg/ingester.TestPushRace.func2()
      /__w/cortex/cortex/pkg/ingester/ingester_test.go:707 +0x364

Previous read at 0x00c00aafeec0 by goroutine 16613:
  github.com/cortexproject/cortex/pkg/ingester.(*labelSetCounter).backFillLimit()
      /__w/cortex/cortex/pkg/ingester/user_state.go:138 +0x667
  github.com/cortexproject/cortex/pkg/ingester.(*userTSDB).PreCreation.(*labelSetCounter).canAddSeriesForLabelSet.func1()
      /__w/cortex/cortex/pkg/ingester/user_state.go:128 +0x39e
  github.com/cortexproject/cortex/pkg/ingester.(*Limiter).AssertMaxSeriesPerLabelSet()
      /__w/cortex/cortex/pkg/ingester/limiter.go:118 +0x2a4
  github.com/cortexproject/cortex/pkg/ingester.(*labelSetCounter).canAddSeriesForLabelSet()
      /__w/cortex/cortex/pkg/ingester/user_state.go:118 +0x548
  github.com/cortexproject/cortex/pkg/ingester.(*userTSDB).PreCreation()
      /__w/cortex/cortex/pkg/ingester/ingester.go:451 +0x3e9
  github.com/prometheus/prometheus/tsdb.(*stripeSeries).getOrSet()
      /__w/cortex/cortex/vendor/github.com/prometheus/prometheus/tsdb/head.go:2004 +0xa2
  github.com/prometheus/prometheus/tsdb.(*Head).getOrCreateWithID()
      /__w/cortex/cortex/vendor/github.com/prometheus/prometheus/tsdb/head.go:1703 +0x13d
  github.com/prometheus/prometheus/tsdb.(*Head).getOrCreate()
      /__w/cortex/cortex/vendor/github.com/prometheus/prometheus/tsdb/head.go:1699 +0xe4
  github.com/prometheus/prometheus/tsdb.(*headAppender).getOrCreate()
      /__w/cortex/cortex/vendor/github.com/prometheus/prometheus/tsdb/head_append.go:438 +0x4ab
  github.com/prometheus/prometheus/tsdb.(*headAppender).Append()
      /__w/cortex/cortex/vendor/github.com/prometheus/prometheus/tsdb/head_append.go:334 +0x13b
  github.com/prometheus/prometheus/tsdb.(*dbAppender).Append()
      <autogenerated>:1 +0xa1
  github.com/cortexproject/cortex/pkg/ingester.(*Ingester).Push()
      /__w/cortex/cortex/pkg/ingester/ingester.go:1305 +0x3a82
  github.com/cortexproject/cortex/pkg/ingester.TestPushRace.func2()
      /__w/cortex/cortex/pkg/ingester/ingester_test.go:707 +0x364

Goroutine 16642 (running) created at:
  github.com/cortexproject/cortex/pkg/ingester.TestPushRace()
      /__w/cortex/cortex/pkg/ingester/ingester_test.go:705 +0x1224
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1743 +0x44

Goroutine 16613 (running) created at:
  github.com/cortexproject/cortex/pkg/ingester.TestPushRace()
      /__w/cortex/cortex/pkg/ingester/ingester_test.go:705 +0x1224
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1743 +0x44
==================
--- FAIL: TestPushRace (3.77s)
    testing.go:1399: race detected during execution of test

To Reproduce
Steps to reproduce the behavior:

  1. Start Cortex (SHA or version)
  2. Perform Operations(Read/Write/Others)

Expected behavior
A clear and concise description of what you expected to happen.

Environment:

  • Infrastructure: [e.g., Kubernetes, bare-metal, laptop]
  • Deployment tool: [e.g., helm, jsonnet]

Additional Context

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions