Skip to content

panic when distributor ha tracker configure to memberlist #4879

Closed
@songjiayang

Description

@songjiayang

Describe the bug
cortex panic when set distributor's ha_tracker kvstore to memberlist.

To Reproduce
Steps to reproduce the behavior:

  1. Start Cortex (SHA or version)
    quay.io/cortexproject/cortex:master-d4fceb1
  2. run with config file:
auth_enabled: true

limits:
  accept_ha_samples: true

distributor:
  shard_by_all_labels: true
  pool:
    health_check_ingesters: true
  ha_tracker:
    enable_ha_tracker: true
    kvstore:
      store: memberlist

memberlist:
  join_members: [cortex-1, cortex-2 ]

Expected behavior

distributor ha tracker works well for memberlist kv store.

Environment:

  • Infrastructure: [e.g., laptop]
  • Deployment tool: [e.g., docker-compose]

Additional Context

level=info ts=2022-09-26T15:50:32.7455646Z caller=main.go:194 msg="Starting Cortex" version="(version=1.13.0, branch=master, revision=d4fceb1)"
level=info ts=2022-09-26T15:50:32.7463937Z caller=server.go:306 http=[::]:80 grpc=[::]:9095 msg="server listening on addresses"
level=debug ts=2022-09-26T15:50:32.7476913Z caller=api.go:141 msg="api: registering route" methods=GET path=/config auth=false
level=debug ts=2022-09-26T15:50:32.7490185Z caller=api.go:141 msg="api: registering route" methods=GET path=/ auth=false
level=debug ts=2022-09-26T15:50:32.7491497Z caller=api.go:141 msg="api: registering route" methods=GET path=/debug/fgprof auth=false
level=debug ts=2022-09-26T15:50:32.7495848Z caller=api.go:141 msg="api: registering route" methods=GET path=/runtime_config auth=false
level=debug ts=2022-09-26T15:50:32.7505063Z caller=api.go:141 msg="api: registering route" methods=GET path=/memberlist auth=false
level=debug ts=2022-09-26T15:50:32.7519924Z caller=api.go:141 msg="api: registering route" methods=GET,POST path=/ingester/ring auth=false
level=debug ts=2022-09-26T15:50:32.752342Z caller=api.go:141 msg="api: registering route" methods=GET,POST path=/ring auth=false
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x164f776]

goroutine 1 [running]:
github.com/cortexproject/cortex/pkg/ring/kv.createClient({_, _}, {_, _}, {{{0x21421c1, 0xe}, {{0x0, 0x0}}, 0x4a817c800, 0x0, ...}, ...}, ...)
	/__w/cortex/cortex/pkg/ring/kv/client.go:146 +0x496
github.com/cortexproject/cortex/pkg/ring/kv.NewClient({{0xc0008552c0, 0xa}, {0x213bdb5, 0xb}, {{{0x21421c1, 0xe}, {{...}}, 0x4a817c800, 0x0, 0x3ff0000000000000, ...}, ...}, ...}, ...)
	/__w/cortex/cortex/pkg/ring/kv/client.go:123 +0x11e
github.com/cortexproject/cortex/pkg/distributor.newHATracker({0x1, 0x37e11d600, 0x12a05f200, 0x6fc23ac00, {{0xc0008552c0, 0xa}, {0x213bdb5, 0xb}, {{{...}, {...}, ...}, ...}, ...}}, ...)
	/__w/cortex/cortex/pkg/distributor/ha_tracker.go:177 +0x9ac
github.com/cortexproject/cortex/pkg/distributor.New({{0x37e11d600, 0x1, 0x77359400}, {0x1, 0x37e11d600, 0x12a05f200, 0x6fc23ac00, {{0xc0008552c0, 0xa}, {0x213bdb5, ...}, ...}}, ...}, ...)
	/__w/cortex/cortex/pkg/distributor/distributor.go:201 +0x1a7
github.com/cortexproject/cortex/pkg/cortex.(*Cortex).initDistributorService(0xc00067ca00)
	/__w/cortex/cortex/pkg/cortex/modules.go:200 +0x18d
github.com/cortexproject/cortex/pkg/util/modules.(*Manager).initModule(0xc000159068, {0x212e689, 0x3}, 0x1e?, 0x0?)
	/__w/cortex/cortex/pkg/util/modules/modules.go:106 +0x233
github.com/cortexproject/cortex/pkg/util/modules.(*Manager).InitModuleServices(0x1ddc440?, {0xc00063f0c0, 0x1, 0x3?})
	/__w/cortex/cortex/pkg/util/modules/modules.go:78 +0x10c
github.com/cortexproject/cortex/pkg/cortex.(*Cortex).Run(0xc00067ca00)
	/__w/cortex/cortex/pkg/cortex/cortex.go:398 +0x225
main.main()
	/__w/cortex/cortex/cmd/cortex/main.go:196 +0xdd3

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions