Closed
Description
Describe the bug
cortex panic when set distributor's ha_tracker kvstore to memberlist.
To Reproduce
Steps to reproduce the behavior:
- Start Cortex (SHA or version)
quay.io/cortexproject/cortex:master-d4fceb1 - run with config file:
auth_enabled: true
limits:
accept_ha_samples: true
distributor:
shard_by_all_labels: true
pool:
health_check_ingesters: true
ha_tracker:
enable_ha_tracker: true
kvstore:
store: memberlist
memberlist:
join_members: [cortex-1, cortex-2 ]
Expected behavior
distributor ha tracker works well for memberlist kv store.
Environment:
- Infrastructure: [e.g., laptop]
- Deployment tool: [e.g., docker-compose]
Additional Context
level=info ts=2022-09-26T15:50:32.7455646Z caller=main.go:194 msg="Starting Cortex" version="(version=1.13.0, branch=master, revision=d4fceb1)"
level=info ts=2022-09-26T15:50:32.7463937Z caller=server.go:306 http=[::]:80 grpc=[::]:9095 msg="server listening on addresses"
level=debug ts=2022-09-26T15:50:32.7476913Z caller=api.go:141 msg="api: registering route" methods=GET path=/config auth=false
level=debug ts=2022-09-26T15:50:32.7490185Z caller=api.go:141 msg="api: registering route" methods=GET path=/ auth=false
level=debug ts=2022-09-26T15:50:32.7491497Z caller=api.go:141 msg="api: registering route" methods=GET path=/debug/fgprof auth=false
level=debug ts=2022-09-26T15:50:32.7495848Z caller=api.go:141 msg="api: registering route" methods=GET path=/runtime_config auth=false
level=debug ts=2022-09-26T15:50:32.7505063Z caller=api.go:141 msg="api: registering route" methods=GET path=/memberlist auth=false
level=debug ts=2022-09-26T15:50:32.7519924Z caller=api.go:141 msg="api: registering route" methods=GET,POST path=/ingester/ring auth=false
level=debug ts=2022-09-26T15:50:32.752342Z caller=api.go:141 msg="api: registering route" methods=GET,POST path=/ring auth=false
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x164f776]
goroutine 1 [running]:
github.com/cortexproject/cortex/pkg/ring/kv.createClient({_, _}, {_, _}, {{{0x21421c1, 0xe}, {{0x0, 0x0}}, 0x4a817c800, 0x0, ...}, ...}, ...)
/__w/cortex/cortex/pkg/ring/kv/client.go:146 +0x496
github.com/cortexproject/cortex/pkg/ring/kv.NewClient({{0xc0008552c0, 0xa}, {0x213bdb5, 0xb}, {{{0x21421c1, 0xe}, {{...}}, 0x4a817c800, 0x0, 0x3ff0000000000000, ...}, ...}, ...}, ...)
/__w/cortex/cortex/pkg/ring/kv/client.go:123 +0x11e
github.com/cortexproject/cortex/pkg/distributor.newHATracker({0x1, 0x37e11d600, 0x12a05f200, 0x6fc23ac00, {{0xc0008552c0, 0xa}, {0x213bdb5, 0xb}, {{{...}, {...}, ...}, ...}, ...}}, ...)
/__w/cortex/cortex/pkg/distributor/ha_tracker.go:177 +0x9ac
github.com/cortexproject/cortex/pkg/distributor.New({{0x37e11d600, 0x1, 0x77359400}, {0x1, 0x37e11d600, 0x12a05f200, 0x6fc23ac00, {{0xc0008552c0, 0xa}, {0x213bdb5, ...}, ...}}, ...}, ...)
/__w/cortex/cortex/pkg/distributor/distributor.go:201 +0x1a7
github.com/cortexproject/cortex/pkg/cortex.(*Cortex).initDistributorService(0xc00067ca00)
/__w/cortex/cortex/pkg/cortex/modules.go:200 +0x18d
github.com/cortexproject/cortex/pkg/util/modules.(*Manager).initModule(0xc000159068, {0x212e689, 0x3}, 0x1e?, 0x0?)
/__w/cortex/cortex/pkg/util/modules/modules.go:106 +0x233
github.com/cortexproject/cortex/pkg/util/modules.(*Manager).InitModuleServices(0x1ddc440?, {0xc00063f0c0, 0x1, 0x3?})
/__w/cortex/cortex/pkg/util/modules/modules.go:78 +0x10c
github.com/cortexproject/cortex/pkg/cortex.(*Cortex).Run(0xc00067ca00)
/__w/cortex/cortex/pkg/cortex/cortex.go:398 +0x225
main.main()
/__w/cortex/cortex/cmd/cortex/main.go:196 +0xdd3
Metadata
Metadata
Assignees
Labels
No labels