-
-
Notifications
You must be signed in to change notification settings - Fork 33.2k
Closed
Labels
benchmarkIssues and PRs related to the benchmark subsystem.Issues and PRs related to the benchmark subsystem.httpIssues or PRs related to the http subsystem.Issues or PRs related to the http subsystem.
Description
I was working on some HTTP parser improvements and I noticed the numbers from the http/simple benchmark fluctuate wildly, even when the system is otherwise quiescent. This is with master, without my changes applied.
Results from 10 runs. Columns are mean(x) median(x) std(x) std(x)/median(x)*100
.
http/simple.js c=50 chunks=0 length=4 type="bytes": 6041.3 6201.2 625.4 10.1%
http/simple.js c=500 chunks=0 length=4 type="bytes": 4706.7 4731.9 264.0 5.6%
http/simple.js c=50 chunks=1 length=4 type="bytes": 5683.0 5600.1 289.0 5.2%
http/simple.js c=500 chunks=1 length=4 type="bytes": 4435.2 4444.8 164.2 3.7%
http/simple.js c=50 chunks=4 length=4 type="bytes": 1157.7 1161.3 13.1 1.1%
http/simple.js c=500 chunks=4 length=4 type="bytes": 2797.2 2835.9 145.1 5.1%
http/simple.js c=50 chunks=0 length=1024 type="bytes": 5950.9 6013.6 318.8 5.3%
http/simple.js c=500 chunks=0 length=1024 type="bytes": 4648.4 4601.7 346.3 7.5%
http/simple.js c=50 chunks=1 length=1024 type="bytes": 5494.6 5365.3 297.9 5.6%
http/simple.js c=500 chunks=1 length=1024 type="bytes": 4367.5 4403.7 375.8 8.5%
http/simple.js c=50 chunks=4 length=1024 type="bytes": 1159.9 1161.8 6.8 0.6%
http/simple.js c=500 chunks=4 length=1024 type="bytes": 2772.3 2744.5 90.9 3.3%
http/simple.js c=50 chunks=0 length=102400 type="bytes": 1587.4 1582.8 28.9 1.8%
http/simple.js c=500 chunks=0 length=102400 type="bytes": 1458.2 1450.0 36.9 2.5%
http/simple.js c=50 chunks=1 length=102400 type="bytes": 952.4 958.8 27.1 2.8%
http/simple.js c=500 chunks=1 length=102400 type="bytes": 864.7 877.1 43.3 4.9%
http/simple.js c=50 chunks=4 length=102400 type="bytes": 1560.2 1612.4 121.2 7.5%
http/simple.js c=500 chunks=4 length=102400 type="bytes": 1454.6 1444.4 102.7 7.1%
http/simple.js c=50 chunks=0 length=4 type="buffer": 5817.7 5801.9 193.1 3.3%
http/simple.js c=500 chunks=0 length=4 type="buffer": 4655.4 4726.7 191.8 4.1%
http/simple.js c=50 chunks=1 length=4 type="buffer": 5736.3 5918.6 298.0 5.0%
http/simple.js c=500 chunks=1 length=4 type="buffer": 4478.3 4400.9 394.2 9.0%
http/simple.js c=50 chunks=4 length=4 type="buffer": 5348.3 5386.7 288.5 5.4%
http/simple.js c=500 chunks=4 length=4 type="buffer": 4215.7 4174.4 254.0 6.1%
http/simple.js c=50 chunks=0 length=1024 type="buffer": 5725.0 5748.3 373.2 6.5%
http/simple.js c=500 chunks=0 length=1024 type="buffer": 4503.7 4430.6 417.6 9.4%
http/simple.js c=50 chunks=1 length=1024 type="buffer": 5584.4 5621.3 189.1 3.4%
http/simple.js c=500 chunks=1 length=1024 type="buffer": 4221.0 4170.0 368.7 8.8%
http/simple.js c=50 chunks=4 length=1024 type="buffer": 4897.6 4860.7 397.4 8.2%
http/simple.js c=500 chunks=4 length=1024 type="buffer": 4015.6 3935.0 343.4 8.7%
http/simple.js c=50 chunks=0 length=102400 type="buffer": 4451.8 4370.2 160.1 3.7%
http/simple.js c=500 chunks=0 length=102400 type="buffer": 3323.9 3356.2 209.9 6.3%
http/simple.js c=50 chunks=1 length=102400 type="buffer": 4168.1 4168.7 256.0 6.1%
http/simple.js c=500 chunks=1 length=102400 type="buffer": 3326.4 3310.8 267.8 8.1%
http/simple.js c=50 chunks=4 length=102400 type="buffer": 3930.1 3945.4 202.3 5.1%
http/simple.js c=500 chunks=4 length=102400 type="buffer": 3128.2 3143.2 139.6 4.4%
Everything with a standard deviation > 1-2% is so unreliable as to be worthless, IMO.
Can someone confirm whether they are seeing similar fluctuations?
cc @nodejs/benchmarking
Metadata
Metadata
Assignees
Labels
benchmarkIssues and PRs related to the benchmark subsystem.Issues and PRs related to the benchmark subsystem.httpIssues or PRs related to the http subsystem.Issues or PRs related to the http subsystem.