Unable to complete query with a single unavailable store-gateway, with shuffle-sharding and zone-awareness

[roystchiang]

Describe the bug
Queriers do not seem to gracefully handle store-gateway failure, even with zone-awareness, and replication.

I have 3 different store-gateways, each in different AZs, and a replication factor of 3. If queriers loses network connection to store-gateway, or store-gateway fails to response due to any failure, queriers will respond with 5xx.

I'm wondering if this is the expected failure mode, or a misconfiguration on my side.

in

cortex/pkg/querier/blocks_store_queryable.go

Line 694 in 464c424

if err := g.Wait(); err != nil {

, it seems like if we fail to query from any store-gateway, we'll fail the call, even if we have successfully gathered blocks from other store-gateways. Since our replication factor is 3, I expect querier to indeed query from all store-gateways, but a single store-gateway failure should not result in 5xx on query.

if we are failing a query because a single store-gateway failed, can we instead return the blocks that were queried, and have queryWithConsistencyCheck

cortex/pkg/querier/blocks_store_queryable.go

Line 503 in 464c424

for attempt := 1; attempt <= maxFetchSeriesAttempts; attempt++ {

reattempt the missing blocks?

To Reproduce
Steps to reproduce the behavior:

1. Start Cortex (3291733)
2. Configure store-gateway with shuffle-sharding and zone-awareness

store_gateway:
  sharding_enabled: true
  sharding_ring:
    replication_factor: 3
    zone_awareness_enabled: true
    instance_availability_zone: AZ

3. scale up store-gateway to 3 pods
4. cut connection between one of the store-gateways and all queriers
5. make a query
6. query fails with the following error

{"status":"error","errorType":"internal","error":"expanding series: failed to fetch series from IP:Port: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp IP:Port: i/o timeout""}

Expected behavior
I expect queries to succeed instead of failing, since store-gateay is zone-aware, and that the data should be replicated to 3 instances. Any single store-gateway failure shouldn't fail the query.

Environment:

• Infrastructure: kubernetes
• Deployment tool: helm
• AZs: 3 availability zones
• Number of store-gateway: 3

Storage Engine

• Blocks
• Chunks

Additional Context

• I used chaos-mesh to help with simulating network failure

[pracucci]

if we fail to query from any store-gateway, we'll fail the call

That's correct. Right now, if a request to any store-gateway fails, the query will fail (then we have a retry mechanism in query-frontend for range queries, but that's at a different level and unfortunately retried query could still fail). This is less than ideal and I agree something to work on.

I expect querier to indeed query from all store-gateways

Well, a querier doesn't necessarily touch all store-gateways. We try to query each block from 1 store-gateway only. If consistency check fail, than we retry the missing blocks from other store-gateways.

if we are failing a query because a single store-gateway failed, can we instead return the blocks that were queried, and have queryWithConsistencyCheck reattempt the missing blocks?

Yes, I think that could be a good solution.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[roystchiang]

it's still impacting

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[friedrich-at-adobe]

Still valid