Bug: POST query got query no expression error if vertical sharding is enabled.

[yeya24]

Describe the bug

The bug happens when vertical sharding is enabled for the tenant by setting shard size > 1. At that time, if we do a POST /api/v1/query query with a non shardable query, such as up, we will always hit errors like.

{
    "status": "error",
    "errorType": "bad_data",
    "error": "invalid parameter \"query\": 1:1: parse error: no expression found in input"
}

I debugged the problem and my findings are:

https://github.com/cortexproject/cortex/blob/master/pkg/frontend/transport/handler.go#L138, we wrap the request body with max body reader here and I think we can only read it once.

	// Buffer the body for later use to track slow queries.
	var buf bytes.Buffer
	r.Body = http.MaxBytesReader(w, r.Body, f.cfg.MaxBodySize)
	r.Body = io.NopCloser(io.TeeReader(r.Body, &buf))

If we enabled vertical sharding for the tenant, we will parse the body and get the query param here https://github.com/cortexproject/cortex/blob/master/pkg/querier/tripperware/roundtrip.go#L153. That's why the issue only happens when shard size > 1. Here query := r.FormValue("query") calls io.ReadAll on the body reader and gets it into EOF.

If query is not shardable we will call next.RoundTrip(), which calls https://github.com/cortexproject/cortex/blob/master/pkg/frontend/transport/roundtripper.go#L37. Here we need to call io.ReadAll on the body again but since we already read the body so this time it returns empty body.

The request gets into the QFE queue and querier will see request with empty body, thus throwing 400 no expression

To Reproduce
Steps to reproduce the behavior:

1. Start Cortex with vertical sharding -frontend.query-stats-enabled=2
2. Run an instant query with POST request that is not shardable, such as up.

Expected behavior
No error will happen