-
Notifications
You must be signed in to change notification settings - Fork 832
Description
Describe the bug
When protobuf codec is enabled, querier will get panic when trying to encode a native histogram with empty bucket.
This is because querier pre-allocates histogram buckets using the expected bucket length of NegativeBuckets
+ PositiveBuckets
+ ZeroBuckets
.
https://github.com/cortexproject/cortex/blob/master/pkg/querier/codec/protobuf_codec.go#L137
However, when a bucket has 0 count it is skipped causing some histogram buckets being nil. https://github.com/cortexproject/cortex/blob/master/pkg/querier/codec/protobuf_codec.go#L214
When marshaling because histogram bucket is expected to be not nil, it caused a panic.
I am able to reproduce the issue using the unit test below.
--- FAIL: TestProtobufCodec_Encode (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x18 pc=0x10510305c]
goroutine 50 [running]:
testing.tRunner.func1.2({0x1054aa1c0, 0x105e86860})
/Users/benye/sdk/go1.23.1/src/testing/testing.go:1632 +0x1bc
testing.tRunner.func1()
/Users/benye/sdk/go1.23.1/src/testing/testing.go:1635 +0x334
panic({0x1054aa1c0?, 0x105e86860?})
/Users/benye/sdk/go1.23.1/src/runtime/panic.go:785 +0x124
github.com/cortexproject/cortex/pkg/querier/tripperware.(*HistogramBucket).MarshalToSizedBuffer(0x140009821f0?, {0x140006aa915?, 0x1?, 0x14000995290?})
/Users/benye/hub/gowork/cortex/pkg/querier/tripperware/query.pb.go:2371 +0x1c
github.com/cortexproject/cortex/pkg/querier/tripperware.(*SampleHistogram).MarshalToSizedBuffer(0x14000995298, {0x140006aa915, 0x27, 0x27})
/Users/benye/hub/gowork/cortex/pkg/querier/tripperware/query.pb.go:2325 +0x1a4
github.com/cortexproject/cortex/pkg/querier/tripperware.(*SampleHistogramPair).MarshalToSizedBuffer(0x14000995290, {0x140006aa915, 0x27, 0x1?})
/Users/benye/hub/gowork/cortex/pkg/querier/tripperware/query.pb.go:2285 +0x34
github.com/cortexproject/cortex/pkg/querier/tripperware.(*SampleStream).MarshalToSizedBuffer(0x14000117310, {0x140006aa915, 0x27, 0x27})
/Users/benye/hub/gowork/cortex/pkg/querier/tripperware/query.pb.go:2222 +0x3e4
github.com/cortexproject/cortex/pkg/querier/tripperware.(*Matrix).MarshalToSizedBuffer(0x14000992678, {0x140006aa915, 0x27, 0x27})
/Users/benye/hub/gowork/cortex/pkg/querier/tripperware/query.pb.go:2771 +0xb0
github.com/cortexproject/cortex/pkg/querier/tripperware.(*PrometheusQueryResult_Matrix).MarshalToSizedBuffer(0x140008498a8?, {0x140006aa915, 0x27, 0x0?})
/Users/benye/hub/gowork/cortex/pkg/querier/tripperware/query.pb.go:2626 +0x38
github.com/cortexproject/cortex/pkg/querier/tripperware.(*PrometheusQueryResult_Matrix).MarshalTo(0x140009821f8, {0x140006aa915, 0x140006a6820?, 0x27})
/Users/benye/hub/gowork/cortex/pkg/querier/tripperware/query.pb.go:2619 +0x48
github.com/cortexproject/cortex/pkg/querier/tripperware.(*PrometheusQueryResult).MarshalToSizedBuffer(0x140006902a0, {0x140006aa900, 0x3c, 0x3c})
/Users/benye/hub/gowork/cortex/pkg/querier/tripperware/query.pb.go:2575 +0xa4
github.com/cortexproject/cortex/pkg/querier/tripperware.(*PrometheusData).MarshalToSizedBuffer(0x14000690290, {0x140006aa900, 0x3c, 0x3c})
/Users/benye/hub/gowork/cortex/pkg/querier/tripperware/query.pb.go:2084 +0x110
github.com/cortexproject/cortex/pkg/querier/tripperware.(*PrometheusResponse).MarshalToSizedBuffer(0x14000690280, {0x140006aa900, 0x3c, 0x3c})
/Users/benye/hub/gowork/cortex/pkg/querier/tripperware/query.pb.go:2032 +0x1c0
github.com/cortexproject/cortex/pkg/querier/tripperware.(*PrometheusResponse).XXX_Marshal(0x105e8a510?, {0x140006aa900, 0x12add4328?, 0x3c}, 0xb0?)
/Users/benye/hub/gowork/cortex/pkg/querier/tripperware/query.pb.go:57 +0x30
github.com/gogo/protobuf/proto.Marshal({0x10561c020, 0x14000690280})
/Users/benye/hub/gowork/cortex/vendor/github.com/gogo/protobuf/proto/table_marshal.go:2940 +0xc0
github.com/cortexproject/cortex/pkg/querier/codec.ProtobufCodec.Encode({0x28?}, 0x70?)
/Users/benye/hub/gowork/cortex/pkg/querier/codec/protobuf_codec.go:46 +0x60
github.com/cortexproject/cortex/pkg/querier/codec.(*InstrumentedCodec).Encode(0x14000849da0, 0x14000117270?)
/Users/benye/hub/gowork/cortex/pkg/querier/codec/instrumented_codec.go:45 +0x30
github.com/cortexproject/cortex/pkg/querier/codec.TestProtobufCodec_Encode.func1(0x14000902d00)
/Users/benye/hub/gowork/cortex/pkg/querier/codec/protobuf_codec_test.go:472 +0x168
testing.tRunner(0x14000902d00, 0x14000995020)
/Users/benye/sdk/go1.23.1/src/testing/testing.go:1690 +0xe4
created by testing.(*T).Run in goroutine 49
/Users/benye/sdk/go1.23.1/src/testing/testing.go:1743 +0x314
To Reproduce
Steps to reproduce the behavior:
- Start Cortex (SHA or version)
- Perform Operations(Read/Write/Others)
Expected behavior
A clear and concise description of what you expected to happen.
Environment:
- Infrastructure: [e.g., Kubernetes, bare-metal, laptop]
- Deployment tool: [e.g., helm, jsonnet]
Additional Context