Set TCP_NODELAY on TCP transports by default #373

1st1 · 2016-07-05T23:29:57Z

This PR enables TCP_NODELAY for all TCP transports.

gvanrossum · 2016-07-05T23:40:57Z

Shouldn't we at least have a way to turn it off? Or why do sockets not have this on by default?

jimfulton · 2016-07-05T23:42:56Z

Assuming that appveyor eventually goes green, LGTM. Thanks!

1st1 · 2016-07-06T00:03:51Z

@gvanrossum Please see #286, where I wanted to add another base Transport class - TCPTransport, with a set_nodelay method. I still think it'd be a good idea ;)

Or why do sockets not have this on by default?

NODELAY can be a bad thing in cases where you want to send as few packages as possible. Setting it causes writes to be sent asap, instead of waiting until the buffer is full or for a TCP ACK.

Without TCP_NODELAY socket operations can have a huge latency. It's particularly important to set for database drivers, HTTP servers and clients etc, where we want the latency to be as small as possible. For modern applications always setting TCP_NODELAY is a good default (and, for instance, that's what Golang does for all TCP connections).

Martiusweb · 2016-07-06T08:31:45Z

If TCP_NODELAY is enabled, a call to write()/send() means that a TCP packet is emitted regardless of if the client ACK'ed the previous packet (as long as the congestion window isn't full), while when disabled, the packet is emited only when there is as much as the MTU to send (or after a well known delay, usually 200ms).

This is a full win when the user sends a whole message (as seen by the application protocol) in one call, and for "synchronized" protocols such as http when peers don't usually write on the socket "at the same time".

Regarding the PR, isn't TCP_NODELAY available on windows/with proactor?

1st1 · 2016-07-06T15:57:55Z

If TCP_NODELAY is enabled, a call to write()/send() means that a TCP packet is emitted regardless of if the client ACK'ed the previous packet (as long as the congestion window isn't full), while when disabled, the packet is emited only when there is as much as the MTU to send (or after a well known delay, usually 200ms).

I'm curious what happens when you use os.writev() call – writing a bunch of buffers with one syscall. Will a bunch of small buffers sent with writev be aggregated in one TCP packet?

Regarding the PR, isn't TCP_NODELAY available on windows/with proactor?

Yes, good catch. I'll add it to the proactor too.

1st1 · 2016-07-06T16:15:04Z

If TCP_NODELAY is enabled, a call to write()/send() means that a TCP packet is emitted regardless of if the client ACK'ed the previous packet (as long as the congestion window isn't full), while when disabled, the packet is emited only when there is as much as the MTU to send (or after a well known delay, usually 200ms).

I'm curious what happens when you use os.writev() call – writing a bunch of buffers with one syscall. Will a bunch of small buffers sent with writev be aggregated in one TCP packet?

I think I found the answer -- writev gathers the output and transfers the data in a single operation. This is important because we want to start using it when #339 is merged.

Martiusweb · 2016-07-06T16:26:21Z

Only one packet is sent, even with TCP_NODELAY (tested with Linux 4.6.3 and wireshark).

socketpair · 2016-07-06T20:02:13Z

asyncio/selector_events.py

+        # Disable the Nagle algorithm -- small writes will be
+        # sent without waiting for the TCP ACK.  This generally
+        # decreases the latency (in some cases significantly.)
+        _set_nodelay(self._sock)


This will work only for TCP sockets, and not for UNIX stream sockets...

Good catch. We need to add functional unittests for TCP/UNIX transports to catch these sort of errors...

Also, accepted sockets should be set as NODELAY too. Please check that this is true.

Alright. I'll prepare a new patch some time later. Thanks for reviewing this iteration!

See also:
aio-libs/aiohttp#664

Good catch. We need to add functional unittests for TCP/UNIX transports to catch these sort of errors...

Tests probably will show nothing, since error is swallowed. Or, tests should test that TCP_NODELAY has been actually set.

1st1 · 2016-09-12T15:52:13Z

@socketpair I don't care that much about "performance" here. Questions like "what is faster to build, set or tuple" should only be addressed when you have an extremely tight and performance critical loop in your algorithm. Even then your decision will be very CPython specific. Anyways, I like the code as it is now.

sethmlarson · 2016-09-12T16:04:42Z

@1st1 Sorry, misunderstood the usage of _set_nodelay, I thought it was applied to each socket created by asyncio, not just server sockets. If that's the case then how it is now is fine.

1st1 · 2016-09-12T16:47:31Z

@1st1 Sorry, misunderstood the usage of _set_nodelay, I thought it was applied to each socket created by asyncio, not just server sockets. If that's the case then how it is now is fine.

@SethMichaelLarson @socketpair Hm, NODELAY should be applied to client connections & server connections created with loop.create_connection & loop.create_server. Transports created by those functions are instances of _SelectorSocketTransport which applies NODELAY in its constructor, so all of them have the flag set. Am I missing something?

sethmlarson · 2016-09-12T16:50:15Z

@1st1 Oh, so it applies to all sockets as I thought previously. Disregard my above comment. If that's the case then I think I'm still +1 on not using a tuple/set to check socket.family just because of how often it's used.

asvetlov · 2016-09-12T17:00:04Z

I believe the check for set is nothing comparing to syscall.
But using TCP_NODELAY makes sense for almost all use cases and should be enabled by default.

AndreLouisCaron · 2017-02-14T15:37:03Z

@1st1

Yes, good catch. I'll add it to the proactor too.

I just installed Python 3.6 and seems like the proactor doesn't use TCP_NODELAY. Was there some kind of limitation that prevented adding it? If it's a simple omission, would you be open to a PR that adds it there too?

AndreLouisCaron · 2017-02-14T15:41:40Z

For context, I opened issue aio-libs/aiomysql#149 because it has its own switch for TCP_NODELAY (presumably for Python 3.4 and 3.5) that breaks when run against the proactor event loop.

It would be neat if we could have this fixed upstream and if we can remove that broken code from aiomysql.

socketpair · 2017-02-15T06:00:46Z

asyncio/selector_events.py

@@ -640,6 +651,11 @@ def __init__(self, loop, sock, protocol, waiter=None,
        self._eof = False
        self._paused = False

+        # Disable the Nagle algorithm -- small writes will be
+        # sent without waiting for the TCP ACK.  This generally


No, TCP_NODELAY is not connected with ACK. Actually, when nodelay is NOT set, kernel will delay sending small TCP packet, waiting for possible additional data to form one big packet. Since we buffer data in our own way, we already concatenate sequence of small writes to big one. So, kernel never sees sequence of small writes and therefore it is not needed to wait for data to concatenate in kernel.

@socketpair see https://en.wikipedia.org/wiki/Nagle%27s_algorithm
nagle waits ACK or enough data in send buffer. That's why delayed ACK + nagle cause trouble.

You have said only part of algorightm (that is connected with ACK). Much more important thing is what I tried to describe (that is connected with writes < MSS).

P.S. I'm not right too. It's best not to describe Nagle's algorithm in the comment.

socketpair reviewed Jul 6, 2016
View reviewed changes

1st1 force-pushed the nodelay branch 2 times, most recently from 7db527f to bea3a42 Compare September 12, 2016 01:36

Set TCP_NODELAY on TCP transports by default

bea3a42

1st1 merged commit bea3a42 into python:master Sep 12, 2016

AndreLouisCaron mentioned this pull request Feb 14, 2017

Fails and hangs with ProactorEventLoop aio-libs/aiomysql#149

Closed

socketpair reviewed Feb 15, 2017

View reviewed changes

Uh oh!

Set TCP_NODELAY on TCP transports by default #373

Set TCP_NODELAY on TCP transports by default #373

Uh oh!

Conversation

1st1 commented Jul 5, 2016

Uh oh!

gvanrossum commented Jul 5, 2016

Uh oh!

jimfulton commented Jul 5, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

1st1 commented Jul 6, 2016

Uh oh!

Martiusweb commented Jul 6, 2016

Uh oh!

1st1 commented Jul 6, 2016

Uh oh!

1st1 commented Jul 6, 2016

Uh oh!

Martiusweb commented Jul 6, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

socketpair Jul 7, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

1st1 commented Sep 12, 2016

Uh oh!

sethmlarson commented Sep 12, 2016

Uh oh!

1st1 commented Sep 12, 2016

Uh oh!

sethmlarson commented Sep 12, 2016

Uh oh!

asvetlov commented Sep 12, 2016

Uh oh!

AndreLouisCaron commented Feb 14, 2017

Uh oh!

AndreLouisCaron commented Feb 14, 2017

Uh oh!

socketpair Feb 15, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jimfulton commented Jul 5, 2016 •

edited

Loading

socketpair Jul 7, 2016 •

edited

Loading

socketpair Feb 15, 2017 •

edited

Loading