querier.worker-parallelism should be derivable

[owen-d]

Currently, querier.worker-parallelism is the number of parallel queries that can be processed per querier per frontend. In an effort to simplify operating cortex, this could be changed to a max parallelism configuration per querier irrespective of the number of frontends. Changing this would alleviate the common practice of determining querier parallelism -> dividing by n_frontends -> setting querier.worker-parallelism.

Additionally, the queriers already have a DNS watch loop running in order to add/drop ephemeral frontends, which can be used to readjust the derived per-frontend limits.

To maintain backwards compatibility, this change could be effected by adding a new flag and marking querier.worker-parallelism deprecated if necessary.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[joe-elliott]

I would like to take this issue on. I propose:

in the watchDNSLoop after a query frontend is added or removed:

• if total query frontends > worker-parallelism
  • create one worker for each query frontend
• else
  • randomize the order of the query frontends and assign one goroutine to each query frontend in order, starting over at the beginning of the list until there are max-parallelism worker goroutines.

Do we feel we need to add a new flag and deprecate the old? or is it enough to mention this in the release notes?

[pracucci]

The proposed solution LGTM. In the doc should just make clear the case "if total query frontends > worker-parallelism" because we'll effectively overprovision the worker, but looks the safest approach to me.

Do we feel we need to add a new flag and deprecate the old? or is it enough to mention this in the release notes?

The query-frontend is considered stable so we have to honor our stability guarantees without introducing any breaking change (but we can deprecate flags). For this reason, I would suggest introduce a new flag and deprecate the previous one.