The TCP SO_SNDBUF/SO_RCVBUF is fixed and too small, decrease throughput greatly

[areyouok]

The SO_SNDBUF/SO_RCVBUF should not be set , the kernel will control it smartly, and have better performance than fixed value in most case.

the specified buffer can not greater than net.core.wmem_max/net.core.rmem_max, but kernal automantic control can increase to a larger value.

This problem decrease throughput especially in below case:

1. this latency between broker and producer(or consumer) is large, especially for consumer
2. the message body size is big

We implements a ping command using rocketmq remoting.
The new version we remove the below code in NettyRemotingClient/NettyRemotingServer

.option(ChannelOption.SO_SNDBUF, nettyClientConfig.getClientSocketSndBufSize())
.option(ChannelOption.SO_RCVBUF, nettyClientConfig.getClientSocketRcvBufSize())

The old version is 4.9.2, the test result is:

The tps of new version is upper to 20x faster than old version.

The kernel config is

net.core.wmem_max = 212992
net.core.rmem_max = 212992
net.ipv4.tcp_rmem = 4096	87380	6291456
net.ipv4.tcp_wmem = 4096	16384	4194304
net.core.wmem_default = 212992
net.core.rmem_default = 212992

in one broker (M/S sync master async flush), one producer send test.

1. 250 byte body: 0.93x old version (the new version is slower slightly)
2. 4000 byte body: 3.85x of old version
3. 16000 byte body: 6.07x of old version
4. 100000 byte body: 7.93x of old version

The old version is 4.9.3-SNAPSHOT in this test. In this test the network latency from producer to broker is 1.61ms.

[areyouok]

Here is an article about this problem (chinese version):

https://www.infoq.cn/article/sfjkj1c5bz2koxsxybho

[areyouok]

in fact we have more data.

• A. remove SO_SNDBUF/SO_RCVBUF/WRITE_BUFFER_WATER_MARK setting
• B. set SO_SNDBUF/SO_RCVBUF, not set WRITE_BUFFER_WATER_MARK (same as 4.9.2)
• C. set SO_SNDBUF/SO_RCVBUF/WRITE_BUFFER_WATER_MARK
• D. remove SO_SNDBUF/SO_RCVBUF, set WRITE_BUFFER_WATER_MARK

The full rocketmq remoting ping test result

Remoting.pdf

[areyouok]

Cluster produce/consume test

• 6 producer and 6 consumer in 6 machines, linux kernel 3.x
• 4 SyncMaster and 4 Slave in 2 machines, linux kernel 4.x
• each broker has 18 queues for test topic
• each producer has 400 threads, sync send
• network latency: 1.61ms

although the old version has poor performance in single tcp connection, but there are 6*4 tcp connections for produce and consume, respectively. So the produce performance of old version is not too bad, but it's consume qps can't keep up with the produce tps.

version	message body size	producer tps	consumer qps
Old	250	180,000*4	130,000*4
Old	4,000	72,000*4	18,000*4
Old	16,000	19,000*4	4,600*4
Old	100,000	3000*4	800*4
New	250	175,000*4	175,000*4
New	4,000	68,000*4	68,000*4
New	16,000	18,000*4	18,000*4
New	100,000	2800*4	2800*4

[areyouok]

Single producer/broker test, no consumer

• 1 producer , linux kernel 3.x
• 1 SyncMaster 1 Slave, linux kernel 4.x
• 18 queues for test topic
• producer has 400 threads, sync send
• network latency: 1.61ms

version	message body size	produce tps
Old	250	88,000
Old	4,000	18,200
Old	16,000	4,860
Old	100,000	750
New	250	82,000
New	4,000	70,000
New	16,000	29,500
New	100,000	5,950

[dongeforever]

Great Catch!