perf: investigate slowdown for using a goroutine for local calls

[petermattis]

#11196 changed local calls to use a goroutine so that such calls do not block the caller and can be cancelled. But this had a surprising impact on performance of single node benchmarks:

name               old time/op  new time/op  delta
KVInsert1_SQL-8     341µs ± 3%   377µs ± 5%  +10.57%   (p=0.000 n=9+10)
KVInsert10_SQL-8    497µs ± 1%   521µs ± 2%   +4.74%    (p=0.000 n=8+9)
KVInsert100_SQL-8  1.49ms ± 1%  1.54ms ± 1%   +2.88%  (p=0.000 n=10+10)
KVUpdate1_SQL-8     510µs ± 2%   579µs ± 1%  +13.49%   (p=0.000 n=10+8)
KVUpdate10_SQL-8    758µs ± 1%   836µs ± 1%  +10.22%   (p=0.000 n=10+8)
KVUpdate100_SQL-8  2.96ms ± 3%  3.02ms ± 2%   +1.92%  (p=0.007 n=10+10)
KVDelete1_SQL-8     459µs ± 4%   513µs ± 4%  +11.74%  (p=0.000 n=10+10)
KVDelete10_SQL-8    664µs ± 1%   736µs ± 1%  +10.77%   (p=0.000 n=10+9)
KVDelete100_SQL-8  2.40ms ± 3%  2.50ms ± 1%   +3.87%   (p=0.000 n=10+8)
KVScan1_SQL-8       215µs ± 3%   227µs ± 1%   +5.62%   (p=0.000 n=10+9)
KVScan10_SQL-8      246µs ± 2%   252µs ± 1%   +2.37%   (p=0.000 n=10+9)
KVScan100_SQL-8     476µs ± 2%   518µs ± 1%   +8.80%  (p=0.000 n=10+10)

I would have expected an across-the-board impact on these benchmarks which are almost always performing either 1PC transactions or read-only operations. Why does KVScan1 only slow down by 12µs while KVDelete1 slows down by 54µs?

[bdarnell]

A shot in the dark: In the past we've noticed it's slow to grow goroutine stacks from their default size (#8544). Maybe that's what's happening here (and the difference is related to how big of a stack each operation needs)