Description
Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (go version
)?
go version go1.10.2 linux/amd64
Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (go env
)?
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/svn/jessemjchen/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/svn/jessemjchen"
GORACE=""
GOROOT="/home/svn/go"
GOTMPDIR=""
GOTOOLDIR="/home/svn/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build019705885=/tmp/go-build"
What did you do?
I write a server in golang,and I use SetReadDeadline/SetWriteDeadline to set a timeout . But when I benchmark ,and use pprof to get cpu pprof ,I get a bad performance because of this two function.
What did you expect to see?
SetReadDeadline/SetWriteDeadline occupy less cpu time.
What did you see instead?
SetReadDeadline/SetWriteDeadline occupy lots of cpu time.
Activity
ianlancetaylor commentedon Jun 5, 2018
Please show us the whole profile and/or show us the code so that we can recreate the problem ourselves. Thanks.
sandyskies commentedon Jun 5, 2018
@ianlancetaylor Can I upload the svg file?
davecheney commentedon Jun 5, 2018
sandyskies commentedon Jun 5, 2018
@ianlancetaylor @davecheney https://github.com/sandyskies/test/blob/master/profile003.svg
valyala commentedon Jun 5, 2018
@sandyskies , could you build and profile the same program under go 1.9 ? This may be related to #15133
The svg file shows the program spends 15% of CPU time in a
send
syscall and 20% of CPU time inconn.SetWriteDeadline
. Thesend
syscall is usually fast, so it's usually OK if its' time share is comparable toconn.SetWriteDeadline
time share.It looks like the program calls
tars/transport.ConnectHandler.send
too frequently (hundreds thousands or even a million calls per second). Probably it would be better optimizing the program by reducing the frequency oftars/transport.ConnectHandler.send
calls. For instance, process requests in batches. Additionally,conn.SetWriteDeadline
may be called on each connection everytimeout / 10
seconds instead of calling it before eachsend
syscall. But even ifconn.SetWriteDeadline
overhead is completely eliminated, the program will run faster only by1/(1-0.2)
= 1.25 times.sandyskies commentedon Jun 5, 2018
@valyala Thanks for the answer. I have run it under go 1.9,and have a worse performance, because of the muti cpu timmer problem in the issue that u mentioned, I have to upgrade to 1.10 . I mentioned in my issue than I am performing a benchmark ,it's a sever and client model, which client send some data to server ,and server simply sends back. transport.ConnectHandler.send is the function that server use to sends the data back to client . Because it is a shot connection ,So I have to use SetDeadline in every socket/connection . Benchmark is use in this case ,so I should not change the program logic I used.
davecheney commentedon Oct 30, 2018
Hello,
I had an opportunity to spend some time with the OP last week and I believe the following benchmark reproduces the issue they are seeing.
On this laptop I see 18.95% of the time spent in
pthread_cond_wait
From the spelunking the OP and I did my interpretation of the trace is thrashing between the goroutine that owns the timer and the main goroutine which is needs to wake the former so it can wrestle control of the lock over the timer wheel.
davecheney commentedon Oct 31, 2018
Some results on a different laptop
/cc @ianlancetaylor @dvyukov
dvyukov commentedon Oct 31, 2018
A bunch of things to improve here. I am on it.
37 remaining items