An error occurred within the plugin when using query-frontend

[SuperMatt]

Describe the bug
An invalid query performed from Grafana against a Cortex Query-Frotned returns the error "An error occurred within the plugin". When performing this query directly against a Cortex Querier or a Prometheus datasource, a correct error is shown. See the below screenshots. The first is the correct response, the second is the incorrect one:

To Reproduce
Steps to reproduce the behavior:

1. Start Cortex (v1.14.1 arm) via the helm chart (2.1.0) with the following components, configured with zone awareness across 3 zones
   a. nginx
   b. distributors
   c. ingesters
   d. store gateways (s3 backend)
   e. queriers
   f. query frontend
   g. query schedulers
   h. ruler
   i. compactor
2. Configure Grafana 9.3.6 with a datasource pointing at either the nginx or query frontends
3. Perform a basic broken query like `count by (job) (up})

Expected behavior
Expected to see bad_data: invalid parameter "query": 1:19: parse error: unexpected character: '}' bad_data: 1:19: parse error: unexpected character: '}'

Environment:

• Infrastructure: Kubernetes
• Deployment tool: Helm

Additional Context
A little bit of testing has shown that the issue only occurs when the Grafana datasource points to a query-frontend, but actually works correctly when pointed straight at a querier.

[SuperMatt]

I've just performed a couple of curls against the query-frontend and queriers and there is one difference which stands out, and that is that the query-frontend appears to be adding this header to the response: Content-Type: text/plain; charset=utf-8. Below you can see the two curl responses. The first one is from the query-frontend which has the Content-Type header, whereas the second one is querier, and it does not have this header.

root@cortex-a-nginx-7d9d96f8ff-r4f8b:/# curl "http://cortex-a-query-frontend:8080/prometheus/api/v1/query?query=up)" -H "X-Scope-OrgID: foobar" -v; echo
*   Trying 172.20.254.26:8080...
* Connected to cortex-a-query-frontend (172.20.254.26) port 8080 (#0)
> GET /prometheus/api/v1/query?query=up) HTTP/1.1
> Host: cortex-a-query-frontend:8080
> User-Agent: curl/7.74.0
> Accept: */*
> X-Scope-OrgID: foobar
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
< Vary: Accept-Encoding
< Date: Thu, 06 Apr 2023 08:03:13 GMT
< Content-Length: 131
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host cortex-a-query-frontend left intact
{"status":"error","errorType":"bad_data","error":"invalid parameter \"query\": 1:4: parse error: unexpected right parenthesis ')'"}


root@cortex-a-nginx-7d9d96f8ff-r4f8b:/# curl "http://cortex-a-querier:8080/prometheus/api/v1/query?query=up)" -H "X-Scope-OrgID: foobar" -v; echo
*   Trying 172.20.225.63:8080...
* Connected to cortex-a-querier (172.20.225.63) port 8080 (#0)
> GET /prometheus/api/v1/query?query=up) HTTP/1.1
> Host: cortex-a-querier:8080
> User-Agent: curl/7.74.0
> Accept: */*
> X-Scope-OrgID: foobar
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
< Content-Type: application/json
< Results-Cache-Gen-Number: 
< Vary: Accept-Encoding
< Date: Thu, 06 Apr 2023 08:03:28 GMT
< Content-Length: 131
< 
* Connection #0 to host cortex-a-querier left intact
{"status":"error","errorType":"bad_data","error":"invalid parameter \"query\": 1:4: parse error: unexpected right parenthesis ')'"}

[SuperMatt]

I've done a whole heap of extra research. I've created an additional proxy in front of the query frontends which changes the header to Content-Type: application/json but the error is still incorrect.

I then upgraded our test grafana instance from 9.3.6 to the latest version: 9.4.7 and instead of seeing the An error occurred within the plugin message, deep diving into the query inspector shows the error message to be correct. However it doesn't show the error inline in the query view.

I then pointed grafana again at the 9.4.7 queriers directly, rather than via the frontends, and the inline error appeared.

All in all this suggests something wrong with the query-frontend which is confusing grafana, but I've yet to figure out what that might be.

9.3.6 - frontend

9.4.7 - frontend

9.4.7 - querier

[yeya24]

Hi @SuperMatt, is this bug reproducible if you use the latest RC?

[SuperMatt]

Hi @yeya24 I can confirm that the problem is still there in the latest RC

[yeya24]

I am not sure why but I cannot reproduce it locally with the latest version. I am just running querier and query frontend binary locally and this is what I got.

[SuperMatt]

That's very odd. I'll have to do some testing tomorrow because I cannot think of any good reason why we would have a difference. I wonder if there's something in my config that's causing this.

[SuperMatt]

I've managed to do an at home test because I had a spare moment, and I was able to replicate this issue.

I used the Cortex Helm Chart in its current iteration and the following configuration for it:

helm install --namespace cortex --set="config.blocks_storage.backend=filesystem" cortex cortex-helm/cortex

I then pointed my Grafana instance in the same namespace directly at the frontend and it gave this error.

This all suggests that there is something that the Cortex Helm Chart might be doing wrong/differently, causing the problem.

The same issue occurs when I upgrade the pods to use the latest rc of cortex itself.

[SuperMatt]

Here's the config diff on the query-frontend in full

alertmanager:
  external_url: /api/prom/alertmanager
api:
  response_compression_enabled: true
auth_enabled: false
blocks_storage:
  backend: filesystem
  bucket_store:
    bucket_index:
      enabled: true
    sync_dir: /data/tsdb-sync
  tsdb:
    dir: /data/tsdb
distributor:
  shard_by_all_labels: true
frontend:
  log_queries_longer_than: 10s
ingester:
  lifecycler:
    join_after: 10s
    num_tokens: 512
    observe_period: 10s
    ring:
      kvstore:
        store: memberlist
ingester_client:
  grpc_client_config:
    max_recv_msg_size: 10485760
    max_send_msg_size: 10485760
limits:
  reject_old_samples: true
  reject_old_samples_max_age: 1w
memberlist:
  join_members:
  - cortex-memberlist
querier:
  active_query_tracker_dir: /data/active-query-tracker
  store_gateway_addresses: dns+cortex-store-gateway-headless:9095
query_range:
  align_queries_with_step: true
  cache_results: true
  results_cache:
    cache:
      memcached:
        expiration: 1h0m0s
      memcached_client:
        timeout: 1s
  split_queries_by_interval: 24h0m0s
ruler:
  enable_api: true
runtime_config:
  file: /etc/cortex-runtime-config/runtime_config.yaml
server:
  grpc_server_max_concurrent_streams: 10000
  grpc_server_max_recv_msg_size: 10485760
  grpc_server_max_send_msg_size: 10485760
  http_listen_port: 8080
target: query-frontend

[SuperMatt]

$ helm upgrade --namespace cortex \
--set="config.blocks_storage.backend=filesystem" \
cortex cortex-helm/cortex \
--set="config.query_range.split_queries_by_interval=0s"  \
--set="config.query_range.cache_results=false

After setting these two options back to their defaults it works correctly.

config.query_range.cache_results=false on its own does not work, config.query_range.split_queries_by_interval=0s also needs to be set. You cannot set that latter option without setting the former.

[yeya24]

@SuperMatt Thanks, I think I can reproduce this now by enabling caching and splitting.

[yeya24]

#5260 This should fix the problem.

[yeya24]

@SuperMatt We got our latest image with the new fix. Can you please let us know, if the fix works for you? Thanks!
https://hub.docker.com/layers/cortexproject/cortex/master-67af431/images/sha256-01c6b8e4b71a780060fb80d73c2b88acc13717a5336e0599d15e1babe411019a?context=explore