net: do not wait on EAGAIN from lossy packet conn

[gopherbot]

by swhite@fitstar.com:

When writing to a connection created by net.Dial("unixgram",
"/dev/log") I will occasionally get an EAGAIN returned.  When that happens the
internal WaitWrite call will block forever if no deadline is set.  I can break by
setting a deadline.  Writing to the connection after the timeout error works so it looks
like the WaitWrite is not working properly for this type of socket.  It happens for both
the Write and the WriteToUnix calls.

I don't know how to create an EAGAIN from a unixgram type socket so I don't have a test
case.  It was very difficult to reproduce this against rsyslog.  I'm hoping a socket
expert can figure it out an easy way to reproduce this.

This bug started on go1.1 and is reproducible on go1.1.1 (never saw it on 1.0.x). 
amd64/linux (ubuntu 12.04)

[alberts]

Comment 1:

Could you post a complete program?

[gopherbot]

Comment 2 by swhite@fitstar.com:

I could not reproduce with any trivial implementation.  The full setup has multiple
server processes all writing to rsyslog and required large number of concurrent requests
before the issue occurred.  I have tried to write several small tests that write to
rsyslog or to a self-opened socket but none have been able to reproduce the lockup.
I guess this is more of a reference in the hopes someone else runs across this with a
simpler setup.

[rsc]

Comment 5:

If we get EAGAIN from a unixgram or udp write, I think we should just return the error.

Labels changed: added priority-later, go1.2, removed priority-triage.

Status changed to Accepted.

[dvyukov]

Comment 6:

>I have tried to write several small tests that write to rsyslog or to a self-opened
socket but none have been able to reproduce the lockup.
Have you tried writing with highest possible speed for a substantial amount of time? You
must be able to send faster than syslog can process. So the buffers must overflow
eventually.

[dvyukov]

Comment 7:

So we don't get epoll notifications for udp/unixgram sockets after EAGAIN?
Returning the error instantly is probably OK for unreliable protocols. But this is a
subtle change in behavior. E.g. a program that checks return codes will crash...

[gopherbot]

Comment 8 by swhite@fitstar.com:

I did a little more research and most of the time, EAGAIN from a unixgram socket works
as expected (does not block forever).  In fact, I can't get it to reproduce with
anything but our full stack test setup:
http load balancer/reverse proxy -> server under test with multiple writers to syslog
all on the same box 
It only takes about 30s of load testing to make it happen in our test environment so it
is 100% reproducible under the right conditions.

[dvyukov]

Comment 9:

Can it be possible that the remote side closes the socket, but we miss the close
notification (rather than write)?

[gopherbot]

Comment 10 by swhite@fitstar.com:

Here are the things I tried that did NOT work to reproduce the bug. FYI, none of these
actually run on the playground since they need file or syslog access.
This code produces an EAGAIN from a unixgram:
http://play.golang.org/p/lf1gs3KGvO
This code writes to rsyslog:
http://play.golang.org/p/lyws2UQj1O
This code uses the library I was using in our code to write to syslog:
http://play.golang.org/p/Am0SmrgY4f
(The current version of the timber library has been patched to avoid the issue entirely
but I disabled that when I tested)

[gopherbot]

Comment 11 by swhite@fitstar.com:

No, the socket is open and ready.  With the timeout set, the code breaks out of the
WaitWrite and I can successfully write to the socket again without any reconnect.
One other piece of information, is that rsyslog is throttling during the test.  But I
don't think that's the cause because it also throttles in both of the examples I just
posted and the EAGAINs don't block forever on those.

[gopherbot]

Comment 12 by swhite@fitstar.com:

Here's a stacktrace of a blocked process:
http://pastebin.com/8exhXBYE
goroutine 4 is the one that's blocked.  The code is part of the
github.com/fitstar/timber library.  There's basically one goroutine draining a channel
and writing to syslog.  A bunch of other goroutines are blocked waiting to write to that
channel and the situation never resolves even with no additional load.

[rsc]

Comment 13:

Labels changed: added go1.2maybe, removed go1.2.

[rsc]

Comment 14:

Too late.

Labels changed: added go1.3maybe, removed go1.2maybe.

[rsc]

Comment 15:

Labels changed: added release-none, removed go1.3maybe.

[rsc]

Comment 16:

Labels changed: added repo-main.

[gopherbot]

Comment 17 by chiparus:

Hello, i'm running into this issue too. 
I have an application using multiple connections to syslog (/dev/log) and with just a
little volume it stalls after only a few messages. Using only one connection this
problem is harder to reproduce.
This program has a 100% reproduction rate for me: http://play.golang.org/p/vu12qIyPil
I tried it running at multiple servers, different rsyslog versions, etc. It doesn't
really matter if rsyslog is configured to drop (rate-limit) messages or not. Even on a
tuned rsyslog config this scripts hangs within 10k messages.
I can send milions of messages repeatedly in one goroutine / one syslog conncetion
without problems. Using 2 goroutines / two syslog connections it hangs within 10k
messages.