node exporter crashes regularly on vm ubuntu 16.04

[ghost]

Hello,
I've set up prometheus with node_exporter on a virtual machine (qemu/kvm) using Ubuntu 16.04. Every 5-6 hours I get the following error:

Apr 02 17:43:51 myhostname.com node_exporter[13406]: time="2018-04-02T17:43:51+03:00" level=info msg="Listening on :9100" source="node_exporter.go:105"
Apr 02 21:44:47 myhostname.com node_exporter[13406]: unexpected fault address 0x0
Apr 02 21:44:47 myhostname.com node_exporter[13406]: fatal error: fault
Apr 02 21:44:47 myhostname.com node_exporter[13406]: [signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x738174]
Apr 02 21:44:47 myhostname.com node_exporter[13406]: goroutine 7 [running]:
Apr 02 21:44:47 myhostname.com node_exporter[13406]: runtime.throw(0xa91696, 0x5)
Apr 02 21:44:47 myhostname.com node_exporter[13406]: /usr/local/go/src/runtime/panic.go:619 +0x81 fp=0xc4204d9528 sp=0xc4204d9508 pc=0x42be91
Apr 02 21:44:47 myhostname.com node_exporter[13406]: runtime.sigpanic()
Apr 02 21:44:47 myhostname.com node_exporter[13406]: /usr/local/go/src/runtime/signal_unix.go:395 +0x211 fp=0xc4204d9578 sp=0xc4204d9528 pc=0x441411
Apr 02 21:44:47 myhostname.com node_exporter[13406]: github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_model/go.(*Counter).GetValue(...)
Apr 02 21:44:47 myhostname.com node_exporter[13406]: /go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_model/go/metrics.pb.go:124
Apr 02 21:44:47 myhostname.com node_exporter[13406]: github.com/prometheus/node_exporter/vendor/github.com/prometheus/common/expfmt.MetricFamilyToText(0xb03880, 0xc4200d04d0, 0xc42014b040
Apr 02 21:44:47 myhostname.com node_exporter[13406]: /go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/common/expfmt/text_create.go:79 +0x264 fp=0xc4204d989
Apr 02 21:44:47 myhostname.com node_exporter[13406]: github.com/prometheus/node_exporter/vendor/github.com/prometheus/common/expfmt.NewEncoder.func4(0xc42014b040, 0x0, 0x0)
Apr 02 21:44:47 myhostname.com node_exporter[13406]: /go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/common/expfmt/encode.go:83 +0x3d fp=0xc4204d98d0 sp=0
Apr 02 21:44:47 myhostname.com node_exporter[13406]: github.com/prometheus/node_exporter/vendor/github.com/prometheus/common/expfmt.encoder.Encode(0xc42040ff20, 0xc42014b040, 0x0, 0x0)
Apr 02 21:44:47 myhostname.com node_exporter[13406]: /go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/common/expfmt/encode.go:36 +0x30 fp=0xc4204d98f8 sp=0
Apr 02 21:44:47 myhostname.com node_exporter[13406]: github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/promhttp.HandlerFor.func1(0x7f967df641f8, 0x
Apr 02 21:44:47 myhostname.com node_exporter[13406]: /go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/promhttp/http.go:142 +0x2ee
Apr 02 21:44:47 myhostname.com node_exporter[13406]: net/http.HandlerFunc.ServeHTTP(0xc420226500, 0x7f967df641f8, 0xc420226730, 0xc4203e8b00)
Apr 02 21:44:47 myhostname.com node_exporter[13406]: /usr/local/go/src/net/http/server.go:1947 +0x44 fp=0xc4204d9a60 sp=0xc4204d9a38 pc=0x670854
Apr 02 21:44:47 myhostname.com node_exporter[13406]: github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1(0x7
Apr 02 21:44:47 myhostname.com node_exporter[13406]: /go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.g
Apr 02 21:44:47 myhostname.com node_exporter[13406]: net/http.HandlerFunc.ServeHTTP(0xc42025c960, 0x7f967df641f8, 0xc420226730, 0xc4203e8b00)
Apr 02 21:44:47 myhostname.com node_exporter[13406]: /usr/local/go/src/net/http/server.go:1947 +0x44 fp=0xc4204d9ad8 sp=0xc4204d9ab0 pc=0x670854
Apr 02 21:44:47 myhostname.com node_exporter[13406]: github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1(0xb0
Apr 02 21:44:47 myhostname.com node_exporter[13406]: /go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.g
Apr 02 21:44:47 myhostname.com node_exporter[13406]: net/http.HandlerFunc.ServeHTTP(0xc42025cb10, 0xb094e0, 0xc42042c000, 0xc4203e8b00)
Apr 02 21:44:47 myhostname.com node_exporter[13406]: /usr/local/go/src/net/http/server.go:1947 +0x44 fp=0xc4204d9b70 sp=0xc4204d9b48 pc=0x670854
Apr 02 21:44:47 myhostname.com node_exporter[13406]: main.handler(0xb094e0, 0xc42042c000, 0xc4203e8b00)
Apr 02 21:44:47 myhostname.com node_exporter[13406]: /go/src/github.com/prometheus/node_exporter/node_exporter.go:67 +0x718 fp=0xc4204d9cf0 sp=0xc4204d9b70 pc=0x8b95e8
Apr 02 21:44:47 myhostname.com node_exporter[13406]: net/http.HandlerFunc.ServeHTTP(0xabe898, 0xb094e0, 0xc42042c000, 0xc4203e8b00)
Apr 02 21:44:47 myhostname.com node_exporter[13406]: /usr/local/go/src/net/http/server.go:1947 +0x44 fp=0xc4204d9d18 sp=0xc4204d9cf0 pc=0x670854
Apr 02 21:44:47 myhostname.com node_exporter[13406]: net/http.(*ServeMux).ServeHTTP(0x1018a20, 0xb094e0, 0xc42042c000, 0xc4203e8b00)
Apr 02 21:44:47 myhostname.com node_exporter[13406]: /usr/local/go/src/net/http/server.go:2337 +0x130 fp=0xc4204d9d58 sp=0xc4204d9d18 pc=0x6724c0

Which stops node_exporter completely and I need to restart it manually (I'm using systemd).
node_exporter version:

node_exporter --version
node_exporter, version 0.16.0-rc.0 (branch: HEAD, revision: 002c1ca)
build user: root@5ff5455ac873
build date: 20180309-15:09:26
go version: go1.10
I'm also using it a a host level on a completely different server and I've got no troubles, but the version is different (20171205-14:50:53, go1.9.2)

[ghost]

I've just realised that this version is not stable (0.16.0) and the one I'm using on the host level on that different server is the stable version (0.15.2). I'm guessing this problem.

[SuperQ]

This is a duplicate of #870

We are going to cut a new build with go 1.10.1 which should fix things.

[brian-brazil]

Are you sure? It's a different error.

[grobie]

I think @brian-brazil is right, this doesn't look related to #870. It might still be related to issues in go1.10, but let's keep this open until then.

[SuperQ]

Sounds good, I didn't look closely enough.

[SuperQ]

I've published v0.16.0-rc.1. This is now built with Go 1.10.1. Please give it a try and see if it fixes the crash.

[lethargosapatheia]

I've tried it on another machine, though, and it seems to be working fine. I see there are quite a few differences between 0.15 and 0.16 as far as functions are concerned.

I know this is off-topic, but I was wondering if anyone could help me to find community support for prometheus/node_exporter and so on. I've been searching for quite a while, and all I could find was a prometheus irc channel where no one answers or says anything. I've got so many questions and there are so many things that I don't understand about how prometheus works and interrogates the kernel and I'm not sure where I can find help.

Thanks

[SuperQ]

The Prometheus IRC channel is quite active, but many of us who work on the project live in Europe, so there is possibly a time zone difference.

There is also the email / google group. https://groups.google.com/forum/#!forum/prometheus-users

[fajfer]

It still crashes all the time, just saying

node_exporter, version 0.16.0 (branch: HEAD, revision: d42bd70f4363dced6b77d8fc311ea57b63387e4f)
  build user:       root@a67a9bc13a69
  build date:       20180515-15:52:42
  go version:       go1.9.6

[discordianfish]

@sjal Are you sure you're hitting this issue not #870?
Can anyone build the node-exporter with go1.11 beta and see if the problem still exists?

[discordianfish]

Closing due to lack of feedback.