Ring lifecycler logs several errors at ingester startup while replaying TSDB WAL

[pracucci]

When the experimental blocks storage is used, the ingester module initialization (initIngester()) may take several time to replay the WAL. Until the module initialization is not completed, the HTTP server loop is not started, so readiness probe requests (GET /ready) will hang. Once the module initialization is completed, all GET /ready requests are processed, but for most of them the deadline has already been reached and they fail as soon as it's checked (apparently the first place hitting it is in the lifecycler CheckReady()).

Example of errors:

level=error ts=2020-01-15T13:43:20.919263167Z caller=lifecycler.go:216 msg="error talking to the KV store" ring=ingester err="Get http://REDACTED:8500/v1/kv/ring?stale=: context canceled"

This is not a real issue, but the error log may deceive the user.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[pracucci]

This has been solved by #2222