Skip to content

[WIP] Refactor cadvisor metrics collection #1024

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from
Closed

[WIP] Refactor cadvisor metrics collection #1024

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
54baa97
Replace cadvisor with #2974 for testing
ehashman Oct 28, 2021
decd941
Update Prometheus client_golang and common libs
ehashman Oct 28, 2021
df7f8f3
Update vendor
ehashman Oct 28, 2021
6ebcdd3
kubelet: update cAdvisor usage for v0.43
bobbypage Nov 9, 2021
25cfe5c
Use BlockingRegistry for cAdvisor container metrics
ehashman Oct 28, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 10 additions & 9 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ require (
github.com/davecgh/go-spew v1.1.1
github.com/dnaeon/go-vcr v1.0.1 // indirect
github.com/docker/distribution v2.7.1+incompatible
github.com/docker/docker v20.10.2+incompatible
github.com/docker/docker v20.10.7+incompatible
github.com/docker/go-connections v0.4.0
github.com/docker/go-units v0.4.0
github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153
Expand All @@ -47,7 +47,7 @@ require (
github.com/golang/mock v1.4.4
github.com/golang/protobuf v1.5.2
github.com/golangplus/testing v0.0.0-20180327235837-af21d9c3145e // indirect
github.com/google/cadvisor v0.39.2
github.com/google/cadvisor v0.42.1-0.20211024204407-cd50a343c965
github.com/google/go-cmp v0.5.5
github.com/google/gofuzz v1.1.0
github.com/google/uuid v1.1.2
Expand All @@ -73,25 +73,26 @@ require (
github.com/openshift/library-go v0.0.0-20210909124717-1c18e732a117
github.com/pkg/errors v0.9.1
github.com/pmezard/go-difflib v1.0.0
github.com/prometheus/client_golang v1.11.0
github.com/prometheus/client_golang v1.11.1-0.20211024130314-8afb1687d291
github.com/prometheus/client_model v0.2.0
github.com/prometheus/common v0.26.0
github.com/prometheus/common v0.29.0
github.com/quobyte/api v0.1.8
github.com/robfig/cron/v3 v3.0.1
github.com/spf13/cobra v1.1.3
github.com/spf13/pflag v1.0.5
github.com/storageos/go-api v2.2.0+incompatible
github.com/stretchr/testify v1.7.0
github.com/vishvananda/netlink v1.1.0
github.com/vishvananda/netns v0.0.0-20200728191858-db3c7e526aae // indirect
github.com/vmware/govmomi v0.20.3
go.etcd.io/etcd/client/pkg/v3 v3.5.0
go.etcd.io/etcd/client/v3 v3.5.0
go.opentelemetry.io/otel/sdk v0.20.0
go.opentelemetry.io/otel/trace v0.20.0
go.opentelemetry.io/proto/otlp v0.7.0
golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83
golang.org/x/net v0.0.0-20210520170846-37e1c6afe023
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d
golang.org/x/net v0.0.0-20210525063256-abc453219eb5
golang.org/x/oauth2 v0.0.0-20210514164344-f6687ab2804c
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
golang.org/x/sys v0.0.0-20210616094352-59db8d763f22
golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d
Expand Down Expand Up @@ -289,7 +290,7 @@ replace (
github.com/gonum/lapack => github.com/gonum/lapack v0.0.0-20181123203213-e4cdc5a0bff9
github.com/gonum/matrix => github.com/gonum/matrix v0.0.0-20181209220409-c518dec07be9
github.com/google/btree => github.com/google/btree v1.0.1
github.com/google/cadvisor => github.com/openshift/google-cadvisor v0.33.2-0.20210825140152-3e7875ab2744
github.com/google/cadvisor => github.com/bwplotka/cadvisor v0.42.1-0.20211024204407-cd50a343c965
github.com/google/go-cmp => github.com/google/go-cmp v0.5.5
github.com/google/gofuzz => github.com/google/gofuzz v1.1.0
github.com/google/martian => github.com/google/martian v2.1.0+incompatible
Expand Down Expand Up @@ -406,9 +407,9 @@ replace (
github.com/pmezard/go-difflib => github.com/pmezard/go-difflib v1.0.0
github.com/posener/complete => github.com/posener/complete v1.1.1
github.com/pquerna/cachecontrol => github.com/pquerna/cachecontrol v0.0.0-20171018203845-0dec1b30a021
github.com/prometheus/client_golang => github.com/prometheus/client_golang v1.11.0
github.com/prometheus/client_golang => github.com/prometheus/client_golang v1.11.1-0.20211024130314-8afb1687d291
github.com/prometheus/client_model => github.com/prometheus/client_model v0.2.0
github.com/prometheus/common => github.com/prometheus/common v0.26.0
github.com/prometheus/common => github.com/prometheus/common v0.29.0
github.com/prometheus/procfs => github.com/prometheus/procfs v0.6.0
github.com/quobyte/api => github.com/quobyte/api v0.1.8
github.com/remyoudompheng/bigfft => github.com/remyoudompheng/bigfft v0.0.0-20170806203942-52369c62f446
Expand Down
14 changes: 6 additions & 8 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,6 @@ github.com/PuerkitoBio/purell v1.1.1/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbt
github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 h1:d+Bc7a5rLufV/sSk/8dngufqelfh6jnri85riMAaF/M=
github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE=
github.com/ajstarks/svgo v0.0.0-20180226025133-644b8db467af/go.mod h1:K08gAheRH3/J6wwsYMMT4xOr94bZjxIelGM0+d/wbFw=
github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d/go.mod h1:rBZYJk541a8SKzHPHnH3zbiI+7dagKZ0cgpgrD7Fyho=
github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY=
github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e h1:QEF07wC0T1rKkctt1RINW/+RMTVmiwxETico2l3gxJA=
github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e/go.mod h1:3U/XgcO3hCbHZ8TKRvWD2dDTCfh9M9ya+I9JpbB7O8o=
Expand All @@ -74,6 +73,8 @@ github.com/blang/semver v3.5.1+incompatible h1:cQNTCjp13qL8KC3Nbxr/y2Bqb63oX6wdn
github.com/blang/semver v3.5.1+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk=
github.com/boltdb/bolt v1.3.1 h1:JQmyP4ZBrce+ZQu0dY660FMfatumYDLun9hBCUVIkF4=
github.com/boltdb/bolt v1.3.1/go.mod h1:clJnj/oiGkjum5o1McbSZDSLxVThjynRyGBgiAx27Ps=
github.com/bwplotka/cadvisor v0.42.1-0.20211024204407-cd50a343c965 h1:Dq8dF1k+jsZWoGje5Oe3R/GXYPf0O7YIzA+SouVw7to=
github.com/bwplotka/cadvisor v0.42.1-0.20211024204407-cd50a343c965/go.mod h1:SjQXcoHjsFgvcg4Nysf6dkHXSO92cPtmKmftD0D1/Sc=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/certifi/gocertifi v0.0.0-20200922220541-2c3bb06c6054 h1:uH66TXeswKn5PW5zdZ39xEwfS9an067BirqA+P4QaLI=
github.com/certifi/gocertifi v0.0.0-20200922220541-2c3bb06c6054/go.mod h1:sGbDF6GwGcLpkNXPUTkMRoywsNa/ol15pxFe6ERfguA=
Expand Down Expand Up @@ -303,7 +304,6 @@ github.com/jonboulle/clockwork v0.2.2 h1:UOGuzwb1PwsrDAObMuhUnj0p5ULPj8V/xJ7Kx9q
github.com/jonboulle/clockwork v0.2.2/go.mod h1:Pkfl5aHPm1nk2H9h0bjmnJD/BcgbGXUBGnn1kMkgxc8=
github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY=
github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y=
github.com/jpillora/backoff v1.0.0/go.mod h1:J/6gKK9jxlEcS3zixgDgUAsiuZ7yrSoa/FX5e0EB2j4=
github.com/json-iterator/go v1.1.11 h1:uVUAXhF2To8cbw/3xN3pxj6kk7TYKs98NIrTqPlMWAQ=
github.com/json-iterator/go v1.1.11/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
Expand Down Expand Up @@ -400,8 +400,6 @@ github.com/openshift/client-go v0.0.0-20210831095141-e19a065e79f7 h1:iKVU5Tga76k
github.com/openshift/client-go v0.0.0-20210831095141-e19a065e79f7/go.mod h1:D6P8RkJzwdkBExQdYUnkWcePMLBiTeCCr8eQIQ7y8Dk=
github.com/openshift/ginkgo v4.7.0-origin.0+incompatible h1:2qD1n/RAnycWMPjYS6MEAUzRmVoF0ql7ozk1ANv8dcM=
github.com/openshift/ginkgo v4.7.0-origin.0+incompatible/go.mod h1:8METQ1gDhl0KW+pGH4c0DIJYEN/ksVCL6hOuHPmXGnk=
github.com/openshift/google-cadvisor v0.33.2-0.20210825140152-3e7875ab2744 h1:myWWE+QcbnUk2yKaL1zZZePAFwgiytkLMkgWQ/doA4c=
github.com/openshift/google-cadvisor v0.33.2-0.20210825140152-3e7875ab2744/go.mod h1:kN93gpdevu+bpS227TyHVZyCU5bbqCzTj5T9drl34MI=
github.com/openshift/library-go v0.0.0-20210909124717-1c18e732a117 h1:oI7m4VUj/4PXK+B7SJS34WcRqUI1P53H90b3VSnGODY=
github.com/openshift/library-go v0.0.0-20210909124717-1c18e732a117/go.mod h1:fKtzrsRXSWMLiBT1SM8cEVT2YyL7ihx/TEuT3gmgFgQ=
github.com/opentracing/opentracing-go v1.1.0/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o=
Expand All @@ -417,12 +415,12 @@ github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZN
github.com/posener/complete v1.1.1/go.mod h1:em0nMJCgc9GFtwrmVmEMR/ZL6WyhyjMBndrE9hABlRI=
github.com/pquerna/cachecontrol v0.0.0-20171018203845-0dec1b30a021 h1:0XM1XL/OFFJjXsYXlG30spTkV/E9+gmd5GD1w2HE8xM=
github.com/pquerna/cachecontrol v0.0.0-20171018203845-0dec1b30a021/go.mod h1:prYjPmNq4d1NPVmpShWobRqXY3q7Vp+80DqgxxUrUIA=
github.com/prometheus/client_golang v1.11.0 h1:HNkLOAEQMIDv/K+04rukrLx6ch7msSRwf3/SASFAGtQ=
github.com/prometheus/client_golang v1.11.0/go.mod h1:Z6t4BnS23TR94PD6BsDNk8yVqroYurpAkEiz0P2BEV0=
github.com/prometheus/client_golang v1.11.1-0.20211024130314-8afb1687d291 h1:oq+NtOQvQclqmCKwxP+HJRILfbgbe8F1tknxRbFPZAY=
github.com/prometheus/client_golang v1.11.1-0.20211024130314-8afb1687d291/go.mod h1:lNmn4rz0M8Psp+sqmtWKbPgVccFYnoPd9O3eJJzuUao=
github.com/prometheus/client_model v0.2.0 h1:uq5h0d+GuxiXLJLNABMgp2qUWDPiLvgCzz2dUR+/W/M=
github.com/prometheus/client_model v0.2.0/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/common v0.26.0 h1:iMAkS2TDoNWnKM+Kopnx/8tnEStIfpYA0ur0xQzzhMQ=
github.com/prometheus/common v0.26.0/go.mod h1:M7rCNAaPfAosfx8veZJCuw84e35h3Cfd9VFqTh1DIvc=
github.com/prometheus/common v0.29.0 h1:3jqPBvKT4OHAbje2Ql7KeaaSicDBCxMYwEJU1zRJceE=
github.com/prometheus/common v0.29.0/go.mod h1:vu+V0TpY+O6vW9J44gczi3Ap/oXXR10b+M/gUGO4Hls=
github.com/prometheus/procfs v0.6.0 h1:mxy4L2jP6qMonqmq+aTtOx1ifVWUgG/TAmntgbh3xv4=
github.com/prometheus/procfs v0.6.0/go.mod h1:cz+aTbrPOrUb4q7XlbU9ygM+/jj0fzG6c1xBZuNvfVA=
github.com/quobyte/api v0.1.8 h1:+sOX1gIlC/OaLipqVZWrHgly9Kh9Qo8OygeS0mWAg30=
Expand Down
2 changes: 1 addition & 1 deletion pkg/kubelet/cadvisor/cadvisor_linux.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ func New(imageFsInfoProvider ImageFsInfoProvider, rootPath string, cgroupRoots [
}

// Create the cAdvisor container manager.
m, err := manager.New(memory.New(statsCacheDuration, nil), sysFs, housekeepingConfig, includedMetrics, http.DefaultClient, cgroupRoots, "")
m, err := manager.New(memory.New(statsCacheDuration, nil), sysFs, housekeepingConfig, includedMetrics, http.DefaultClient, cgroupRoots, nil /* containerEnvMetadataWhiteList */, "" /* perfEventsFile */, time.Duration(0) /*resctrlInterval*/)
if err != nil {
return nil, err
}
Expand Down
19 changes: 8 additions & 11 deletions pkg/kubelet/server/server.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,8 @@ import (
"time"

"github.com/emicklei/go-restful"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
cadvisormetrics "github.com/google/cadvisor/container"
cadvisorapi "github.com/google/cadvisor/info/v1"
cadvisorv2 "github.com/google/cadvisor/info/v2"
Expand Down Expand Up @@ -355,8 +357,8 @@ func (s *Server) InstallDefaultHandlers() {
//lint:ignore SA1019 https://github.com/kubernetes/enhancements/issues/1206
s.restfulCont.Handle(metricsPath, legacyregistry.Handler())

// cAdvisor metrics are exposed under the secured handler as well
r := compbasemetrics.NewKubeRegistry()
// cAdvisor container metrics require a blocking registry
br := prometheus.NewBlockingRegistry()

includedMetrics := cadvisormetrics.MetricSet{
cadvisormetrics.CpuUsageMetrics: struct{}{},
Expand All @@ -372,18 +374,13 @@ func (s *Server) InstallDefaultHandlers() {
// Only add the Accelerator metrics if the feature is inactive
// Note: Accelerator metrics will be removed in the future, hence the feature gate.
if !utilfeature.DefaultFeatureGate.Enabled(features.DisableAcceleratorUsageMetrics) {
includedMetrics.Add(cadvisormetrics.MetricKind(cadvisormetrics.AcceleratorUsageMetrics))
includedMetrics[cadvisormetrics.AcceleratorUsageMetrics] = struct{}{}
}

cadvisorOpts := cadvisorv2.RequestOptions{
IdType: cadvisorv2.TypeName,
Count: 1,
Recursive: true,
}
r.RawMustRegister(metrics.NewPrometheusCollector(prometheusHostAdapter{s.host}, containerPrometheusLabelsFunc(s.host), includedMetrics, clock.RealClock{}, cadvisorOpts))
r.RawMustRegister(metrics.NewPrometheusMachineCollector(prometheusHostAdapter{s.host}, includedMetrics))
br.MustRegisterRaw(metrics.NewPrometheusCollector(prometheusHostAdapter{s.host}, containerPrometheusLabelsFunc(s.host), includedMetrics, clock.RealClock{}))
br.MustRegister(metrics.NewPrometheusMachineCollector(prometheusHostAdapter{s.host}, includedMetrics))
s.restfulCont.Handle(cadvisorMetricsPath,
compbasemetrics.HandlerFor(r, compbasemetrics.HandlerOpts{ErrorHandling: compbasemetrics.ContinueOnError}),
promhttp.HandlerForTransactional(br, promhttp.HandlerOpts{ErrorHandling: promhttp.ContinueOnError}),
)

s.addMetricsBucketMatcher("metrics/resource")
Expand Down
9 changes: 8 additions & 1 deletion staging/src/k8s.io/api/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,17 @@ require (
)

replace (
github.com/google/cadvisor => github.com/openshift/google-cadvisor v0.33.2-0.20210825140152-3e7875ab2744
cloud.google.com/go => cloud.google.com/go v0.54.0
github.com/cespare/xxhash/v2 => github.com/cespare/xxhash/v2 v2.1.1
github.com/containerd/containerd => github.com/containerd/containerd v1.4.4
github.com/containerd/typeurl => github.com/containerd/typeurl v1.0.1
github.com/docker/docker => github.com/docker/docker v20.10.2+incompatible
github.com/google/cadvisor => github.com/bwplotka/cadvisor v0.42.1-0.20211024204407-cd50a343c965
github.com/imdario/mergo => github.com/imdario/mergo v0.3.5
github.com/mattn/go-colorable => github.com/mattn/go-colorable v0.0.9
github.com/onsi/ginkgo => github.com/openshift/ginkgo v4.7.0-origin.0+incompatible
golang.org/x/net => golang.org/x/net v0.0.0-20210520170846-37e1c6afe023
golang.org/x/oauth2 => golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d
k8s.io/api => ../api
k8s.io/apiextensions-apiserver => ../apiextensions-apiserver
k8s.io/apimachinery => ../apimachinery
Expand Down
Loading