Store-Gateway histogram metrics are `NaN`

[harry671003]

Describe the bug

• Some of the store-gateway histogram metrics are NaN.

# HELP cortex_bucket_store_series_data_fetched How many items of a data type in a block were fetched for a single series request.
# TYPE cortex_bucket_store_series_data_fetched summary
cortex_bucket_store_series_data_fetched{component="store-gateway",data_type="chunks",quantile="0.5"} NaN
cortex_bucket_store_series_data_fetched{component="store-gateway",data_type="chunks",quantile="0.95"} NaN
cortex_bucket_store_series_data_fetched{component="store-gateway",data_type="chunks",quantile="0.99"} NaN
cortex_bucket_store_series_data_fetched_sum{component="store-gateway",data_type="chunks"} 1.090387e+06
cortex_bucket_store_series_data_fetched_count{component="store-gateway",data_type="chunks"} 10534
cortex_bucket_store_series_data_fetched{component="store-gateway",data_type="postings",quantile="0.5"} NaN
cortex_bucket_store_series_data_fetched{component="store-gateway",data_type="postings",quantile="0.95"} NaN
cortex_bucket_store_series_data_fetched{component="store-gateway",data_type="postings",quantile="0.99"} NaN
cortex_bucket_store_series_data_fetched_sum{component="store-gateway",data_type="postings"} 7
cortex_bucket_store_series_data_fetched_count{component="store-gateway",data_type="postings"} 10534
cortex_bucket_store_series_data_fetched{component="store-gateway",data_type="series",quantile="0.5"} NaN
cortex_bucket_store_series_data_fetched{component="store-gateway",data_type="series",quantile="0.95"} NaN
cortex_bucket_store_series_data_fetched{component="store-gateway",data_type="series",quantile="0.99"} NaN
cortex_bucket_store_series_data_fetched_sum{component="store-gateway",data_type="series"} 2
cortex_bucket_store_series_data_fetched_count{component="store-gateway",data_type="series"} 10534

To Reproduce
Steps to reproduce the behavior:

• port-forward to a store-gateway pod and go to /metrics endpoint.

[friedrichg]

These are summaries, not histograms.

cortex/pkg/storegateway/bucket_store_metrics.go

Line 218 in ed36a62

data.SendSumOfSummariesWithLabels(out, m.seriesDataFetched, "thanos_bucket_store_series_data_fetched", "data_type")

And I think this is how it was decided no observation should look like
prometheus/client_golang#85

why is this a problem?

[yeya24]

Yeah I think this should is expected. Probably we could think about changing summary metric to histograms as we can't aggregate summary metrics over multiple store gateway instances.

[harry671003]

I was confused because I saw the NaNs and thought this was a bug.

I was tryng to gather information about p99 and p50 series fetched per query from store-gateways. We have these metrics in ingesters. cortex_ingester_queried_series. Would it be useful to have these histogram metrics?

[friedrichg]

We already have the thanos that uses histograms for these, we should fix the cortex code #5177