Isolate queriers per tenant

[gouthamve]

Well, if a big tenant comes along and opens a dashboard with large queries, he has the potential to take out all queriers one by one.

The hashing algorithm described here shows how to mitigate that: https://twitter.com/colmmacc/status/1034492056968736768

Would be great to come up with something similar for protecting queriers.

[tomwilkie]

+1 I think this would be really useful.

The downside is if a big tenant comes along, they can only use 1/nth of the queries, leaving resources wasted. This is particularly bad as queries rates are spiky and not predictable.

This might work better on the write path, where things are more predictable. However this has the risk of making the write distribution much less even, meaning you'll have to provision ingesters for the biggest "shard" and reduce effective utilisation.