Skip to content

Conversation

alanprot
Copy link
Member

@alanprot alanprot commented Apr 12, 2022

What this PR does:

Fix race condition on /series.

ForReplicationSet can still invoke the callback after the quorum has reached and this can cause the metrics map to be write and read at the same time.

=== RUN   TestDistributor_MetricsForLabelMatchers_SingleSlowIngester
fatal error: concurrent map iteration and map write

goroutine 89 [running]:
runtime.throw({0x1efcf2d, 0x0})
	/usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc000a47c28 sp=0xc000a47bf8 pc=0x10383b1
runtime.mapiternext(0xc000a47da0)
	/usr/local/go/src/runtime/map.go:858 +0x4eb fp=0xc000a47c98 sp=0xc000a47c28 pc=0x10112cb
github.com/cortexproject/cortex/pkg/distributor.(*Distributor).MetricsForLabelMatchers(0x0, {0x21c53d8, 0xc0007f13e0}, 0x1ec9e30, 0x4, {0xc000a47ee0, 0x1, 0x1})
	/Users/approtas/workspaces/cortex/pkg/distributor/distributor.go:985 +0x325 fp=0xc000a47e10 sp=0xc000a47c98 pc=0x1bef3e5
github.com/cortexproject/cortex/pkg/distributor.TestDistributor_MetricsForLabelMatchers_SingleSlowIngester(0x1)
	/Users/approtas/workspaces/cortex/pkg/distributor/distributor_test.go:1870 +0x30e fp=0xc000a47f70 sp=0xc000a47e10 pc=0x1c09b8e
testing.tRunner(0xc0001d0d00, 0x1f3d210)
	/usr/local/go/src/testing/testing.go:1259 +0x102 fp=0xc000a47fc0 sp=0xc000a47f70 pc=0x11156c2

Checklist

  • Tests updated
  • [ NA] Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@alanprot alanprot merged commit 6758f1f into cortexproject:master Apr 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants