Introduce query-tee tool

[pracucci]

What this PR does:
While working on the experimental blocks storage, we started having the need to compare chunks vs blocks storage in the real world. To do it, we setup a test cluster (with blocks storage) ingesting the same exact series of another cluster with chunks storage. We now have two parallel clusters with the same series and we have the need to query both clusters and compare performances and results.

Given this need, the idea we got is to add a new tool called query-tee that exposes the Prometheus API read endpoints and, for each request, send the same request to 2+ backends and make them compete.

In this PR:

• Added a new tool query-tee
• Track a metric with response time for each backend

What's out of the scope of this PR:

• Compare results from backends

Which issue(s) this PR fixes:
N/A

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[pstibrany]

I think it's a good first version. I have some comments about how to make it little bit more generic (by honoring backend paths). Also HTTP client configuration seems strange -- I think there are some conflicting options set. It would be good to clarify that.

[pstibrany]

Nice work and thanks for addressing my feedback.

[pstibrany]

Very nice! I haven't noticed any test for different backend path prefixes (say one backend using /prefix1/api/v1/query and other one different prefix), would it be worth having one?

[pracucci]

Thanks for your review @pstibrany!

I haven't noticed any test for different backend path prefixes (say one backend using /prefix1/api/v1/query and other one different prefix), would it be worth having one?

Makes sense. Done!