Skip to content

LoggingConnectionMiddlewareCanBeAddedBeforeAndAfterHttps() is flaky #34561

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dougbu opened this issue Jul 20, 2021 · 4 comments
Closed

LoggingConnectionMiddlewareCanBeAddedBeforeAndAfterHttps() is flaky #34561

dougbu opened this issue Jul 20, 2021 · 4 comments
Assignees
Labels
area-networking Includes servers, yarp, json patch, bedrock, websockets, http client factory, and http abstractions test-failure
Milestone

Comments

@dougbu
Copy link
Contributor

dougbu commented Jul 20, 2021

Failing Test(s)

  • LoggingConnectionMiddlewareCanBeAddedBeforeAndAfterHttps()

Error Message

System.IO.IOException :  Received an unexpected EOF or 0 bytes from the transport stream.

Stacktrace

     at System.Net.Security.SslStream.<FillHandshakeBufferAsync>g__InternalFillHandshakeBufferAsync|187_0[TIOAdapter](TIOAdapter adap, ValueTask`1 task, Int32 minSize) in System.Net.Security.dll:token 0x600030d+0x8b
   at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter) in System.Net.Security.dll:token 0x60002f1+0x8c
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) in System.Net.Security.dll:token 0x60002f0+0x299
   at Microsoft.AspNetCore.Server.Kestrel.InMemory.FunctionalTests.TestTransport.InMemoryHttpClientSlim.GetStream(Stream rawStream, Uri requestUri, Boolean validateCertificate) in /_/src/Servers/Kestrel/test/InMemory.FunctionalTests/TestTransport/InMemoryHttpClientSlim.cs:line 127
   at Microsoft.AspNetCore.Server.Kestrel.InMemory.FunctionalTests.TestTransport.InMemoryHttpClientSlim.GetStringAsync(Uri requestUri, Boolean validateCertificate) in /_/src/Servers/Kestrel/test/InMemory.FunctionalTests/TestTransport/InMemoryHttpClientSlim.cs:line 35
   at Microsoft.AspNetCore.Server.Kestrel.InMemory.FunctionalTests.TestTransport.InMemoryHttpClientSlim.GetStringAsync(String requestUri, Boolean validateCertificate) in /_/src/Servers/Kestrel/test/InMemory.FunctionalTests/TestTransport/InMemoryHttpClientSlim.cs:line 30
   at Microsoft.AspNetCore.Testing.TaskExtensions.TimeoutAfter[T](Task`1 task, TimeSpan timeout, String filePath, Int32 lineNumber) in /_/src/Shared/TaskExtensions.cs:line 90
   at Microsoft.AspNetCore.Server.Kestrel.InMemory.FunctionalTests.LoggingConnectionMiddlewareTests.LoggingConnectionMiddlewareCanBeAddedBeforeAndAfterHttps() in /_/src/Servers/Kestrel/test/InMemory.FunctionalTests/LoggingConnectionMiddlewareTests.cs:line 33
   at Microsoft.AspNetCore.Server.Kestrel.InMemory.FunctionalTests.LoggingConnectionMiddlewareTests.LoggingConnectionMiddlewareCanBeAddedBeforeAndAfterHttps() in /_/src/Servers/Kestrel/test/InMemory.FunctionalTests/LoggingConnectionMiddlewareTests.cs:line 35
--- End of stack trace from previous location ---

Logs

See https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-aspnetcore-refs-heads-main-cecc8bbbfe7d4877b4/InMemory.FunctionalTests--net6.0/console.1372964f.log?sv=2019-07-07 if more context is needed.

 | [0.003s] TestLifetime Information: Starting test LoggingConnectionMiddlewareCanBeAddedBeforeAndAfterHttps at 2021-07-20T00:23:12
 | [0.007s] Microsoft.Extensions.Hosting.Internal.Host Debug: Hosting starting
 | [2.050s] Microsoft.Hosting.Lifetime Information: Now listening on: https://127.0.0.1:0
 | [2.050s] Microsoft.AspNetCore.Hosting.Diagnostics Debug: Loaded hosting startup assembly InMemory.FunctionalTests
 | [2.050s] Microsoft.Hosting.Lifetime Information: Application started. Press Ctrl+C to shut down.
 | [2.050s] Microsoft.Hosting.Lifetime Information: Hosting environment: Production
 | [2.050s] Microsoft.Hosting.Lifetime Information: Content root path: /private/tmp/helix/working/C8A20A4C/w/B1560994/e/
 | [2.050s] Microsoft.Extensions.Hosting.Internal.Host Debug: Hosting started
 | [6.375s] Microsoft.AspNetCore.Server.Kestrel.Connections Debug: Connection id "0HMAB2I8425HF" accepted.
 | [10.059s] Microsoft.AspNetCore.Server.Kestrel.Connections Debug: Connection id "0HMAB2I8425HF" started.
 | [10.342s] Microsoft.AspNetCore.Server.Kestrel.Core.Internal.LoggingConnectionMiddleware Debug: ReadAsync[169]
 |                                                                                                16 03 01 00 A4 01 00 00  A0 03 03 60 F6 17 72 E6   ....¤...  ..`ö.ræ
 |                                                                                                BA 0B D4 DE 65 26 E4 33  10 81 06 EC 46 4E 2E 03   º.ÔÞe&ä3 ...ìFN..
 |                                                                                                AB 36 59 00 EC E1 C2 65  58 0A A2 00 00 2C 00 FF   «6Y.ìáÂe X.¢..,.ÿ
 |                                                                                                C0 2C C0 2B C0 24 C0 23  C0 0A C0 09 C0 08 C0 30   À,À+À$À# À.À.À.À0
 |                                                                                                C0 2F C0 28 C0 27 C0 14  C0 13 C0 12 00 9D 00 9C   À/À(À'À. À.À.....
 |                                                                                                00 3D 00 3C 00 35 00 2F  00 0A 01 00 00 4B 00 00   .=.<.5./ .....K..
 |                                                                                                00 0E 00 0C 00 00 09 6C  6F 63 61 6C 68 6F 73 74   .......l ocalhost
 |                                                                                                00 0A 00 08 00 06 00 17  00 18 00 19 00 0B 00 02   ........ ........
 |                                                                                                01 00 00 0D 00 12 00 10  04 01 02 01 05 01 06 01   ........ ........
 |                                                                                                04 03 02 03 05 03 06 03  00 05 00 05 01 00 00 00   ........ ........
 |                                                                                                00 00 12 00 00 00 17 00  00                        ........ .
 | [17.933s] Microsoft.AspNetCore.Server.Kestrel.Core.Internal.LoggingConnectionMiddleware Debug: WriteAsync[1242]
 |                                                                                                16 03 03 00 59 02 00 00  55 03 03 60 F6 17 7E 4E   ....Y... U..`ö.~N
 |                                                                                                E1 12 DC 6D 40 4A 7C C6  02 01 16 4A B8 B2 30 AD   á.Üm@J|Æ ...J¸²0­
 |                                                                                                01 A2 0F 77 91 DD C1 33  C1 89 2F 20 07 03 83 BC   .¢.w.��3 �./ ...¼
 |                                                                                                11 38 0D BA 24 4C ED C5  FF 30 EB 00 9A A8 E6 F1   .8.º$LíÅ ÿ0ë..¨æñ
 |                                                                                                6C B8 6D 7C F4 D1 FC 92  C6 07 D3 5E C0 30 00 00   l¸m|ôÑü. Æ.Ó^À0..
 |                                                                                                0D 00 00 00 00 FF 01 00  01 00 00 17 00 00 16 03   .....ÿ.. ........
 |                                                                                                03 03 1C 0B 00 03 18 00  03 15 00 03 12 30 82 03   ........ .....0..
 |                                                                                                0E 30 82 01 F6 A0 03 02  01 02 02 10 34 02 16 F3   .0..ö .. ....4..ó
 |                                                                                                98 53 2E AE 4F 57 F5 F4  4C 6F D0 9F 30 0D 06 09   .S.®OWõô Lo�.0...
 |                                                                                                2A 86 48 86 F7 0D 01 01  0B 05 00 30 14 31 12 30   *.H.÷... ...0.1.0
 |                                                                                                10 06 03 55 04 03 0C 09  6C 6F 63 61 6C 68 6F 73   ...U.... localhos
 |                                                                                                74 30 1E 17 0D 32 30 30  32 32 31 31 39 30 31 33   t0...200 22119013
 |                                                                                                37 5A 17 0D 32 35 30 32  32 31 31 39 30 31 33 37   7Z..2502 21190137
 |                                                                                                5A 30 14 31 12 30 10 06  03 55 04 03 0C 09 6C 6F   Z0.1.0.. .U....lo
 |                                                                                                63 61 6C 68 6F 73 74 30  82 01 22 30 0D 06 09 2A   calhost0 .."0...*
 |                                                                                                86 48 86 F7 0D 01 01 01  05 00 03 82 01 0F 00 30   .H.÷.... .......0
 |                                                                                                82 01 0A 02 82 01 01 00  E5 95 2B 82 12 F1 D2 5A   ........ å.+..ñÒZ
 |                                                                                                64 04 AD DA 2B D9 11 81  09 B7 05 82 F9 53 07 83   d.­Ú+Ù.. .·..ùS..
 |                                                                                                61 18 C0 28 53 11 C7 6C  29 88 B6 92 DC 62 BE 4C   a.À(S.Çl ).¶.Üb¾L
 |                                                                                                7E 3A 15 5D AE 4E DD 2E  34 F7 6B B6 F9 BA ED 72   ~:.]®N�. 4÷k¶ùºír
 |                                                                                                9C 99 39 F0 B3 C2 83 8F  83 16 59 EC B7 AC 4B D9   ..9ð³Â.. ..Yì·¬KÙ
 |                                                                                                E7 46 D3 50 69 52 A2 38  67 03 5C 5A 41 14 96 99   çFÓPiR¢8 g.\ZA...
 |                                                                                                7B CD 41 BA 20 49 81 B0  B7 4C AD 98 87 7B 82 04   {�Aº I.° ·L­..{..
 |                                                                                                F2 CE CD 60 2C 28 B7 F9  42 5A DF 74 66 4F 30 D7   òÎ�`,(·ù BZßtfO0×
 |                                                                                                17 1B 04 41 4C 67 5B CE  EC 7D 32 64 BB E7 74 D6   ...ALg[Î ì}2d»çtÖ
 |                                                                                                BB F3 06 11 6C 34 35 3C  75 1C 9A 94 59 A5 02 D0   »ó..l45< u...Y¥.�
 |                                                                                                52 AF 39 E2 6C A3 7C F7  CA 8B 6C A7 14 39 05 80   R¯9âl£|÷ Ê.l§.9..
 |                                                                                                5F 85 7E 15 29 79 A0 F3  8B EE 72 BD D1 90 CF FF   _.~.)y ó .îr½Ñ.�ÿ
 |                                                                                                C4 F5 D6 16 04 0E 38 69  B0 25 F0 10 2C 5C 11 0D   ÄõÖ...8i °%ð.,\..
 |                                                                                                69 C5 50 E3 24 A6 E7 50  9A DE 7F E4 4A 8E 18 97   iÅPã$¦çP .Þ.äJ...
 |                                                                                                A9 E1 BA 77 80 4B 8C 84  2C 28 E3 F1 B5 56 D7 05   ©áºw.K.. ,(ãñµV×.
 |                                                                                                A5 94 B0 79 60 8E 72 D6  59 8B 00 F7 A6 CD 5E 38   ¥.°y`.rÖ Y..÷¦�^8
 |                                                                                                D8 94 A9 47 19 88 76 A9  02 03 01 00 01 A3 5C 30   Ø.©G..v© .....£\0
 |                                                                                                5A 30 0E 06 03 55 1D 0F  01 01 FF 04 04 03 02 04   Z0...U.. ..ÿ.....
 |                                                                                                B0 30 14 06 03 55 1D 11  04 0D 30 0B 82 09 6C 6F   °0...U.. ..0...lo
 |                                                                                                63 61 6C 68 6F 73 74 30  13 06 03 55 1D 25 04 0C   calhost0 ...U.%..
 |                                                                                                30 0A 06 08 2B 06 01 05  05 07 03 01 30 1D 06 03   0...+... ....0...
 |                                                                                                55 1D 0E 04 16 04 14 63  D2 2C A8 20 CD 65 F1 AC   U......c Ò,¨ �eñ¬
 |                                                                                                02 E2 A1 7E 22 0F DE 8A  80 57 BE 30 0D 06 09 2A   .â¡~".Þ. .W¾0...*
 |                                                                                                86 48 86 F7 0D 01 01 0B  05 00 03 82 01 01 00 A6   .H.÷.... .......¦
 |                                                                                                D1 73 CA A3 2F 77 78 A4  69 BB C7 81 7F 2D 22 2D   ÑsÊ£/wx¤ i»Ç..-"-
 |                                                                                                46 8E 0D 2F C3 D2 48 3F  45 93 55 61 FB 6E 7E 9D   F../ÃÒH? E.Uaûn~.
 |                                                                                                1C 56 5A 9A E9 CD D3 DD  AD 7E AB 6C C5 36 78 85   .VZ.é�Ó� ­~«lÅ6x.
 |                                                                                                B0 97 F2 01 B4 44 AF E6  3C C0 67 A4 D8 35 50 76   °.ò.´D¯æ <Àg¤Ø5Pv
 |                                                                                                DD E5 13 6B BB DA 93 F2  E9 DB EE D7 4C 73 EA 3A   �å.k»Ú.ò éÛî×Lsê:
 |                                                                                                66 D7 DC 2A 9D 70 F1 00  6B 2B 92 F3 9E 57 36 E6   f×Ü*.pñ. k+.ó.W6æ
 |                                                                                                2C AF 7A 99 2A 71 1F F3  1B CF 2E A5 66 E8 1B 27   ,¯z.*q.ó .�.¥fè.'
 |                                                                                                BD 97 D5 5C E3 95 22 C5  EF 58 8C 62 C7 94 73 E3   ½.Õ\ã."Å ïX.bÇ.sã
 |                                                                                                4E 22 BE DE F1 81 1E CF  14 45 9E B5 B1 0F B9 92   N"¾Þñ..� .E.µ±.¹.
 |                                                                                                E0 46 DE 9C A5 DF 3B B0  1A A8 2F 1E E7 D1 CF EC   à FÞ.¥ß;° .¨/.çÑ�ì
 |                                                                                                FD A8 50 91 99 37 8E 58  6D BB 06 1F BE 26 FA D1   ý¨P..7.X m»..¾&úÑ
 |                                                                                                7A 84 49 7F E1 0C 98 1A  58 08 01 33 C9 2E C8 0A   z.I.á... X..3É.È.
 |                                                                                                57 82 06 E4 AC C6 E9 A4  04 F5 2B 2D 63 A4 D3 40   W..ä¬Æé¤ .õ+-c¤Ó@
 |                                                                                                76 09 94 C2 BB DC 1F AE  CD 2D 7B 26 87 A9 C1 76   v..»Ü.® �-{&.©�v
 |                                                                                                0B E8 C2 3C B0 7B 9C BD  9D 18 BC 12 67 10 CA A0   .èÂ<°{.½ ..¼.g.Ê 
 |                                                                                                AD 85 7C 01 96 3F E2 93  B4 6D 57 6A 5D B7 72 16   ­.|..?â. ´mWj]·r.
 |                                                                                                03 03 01 4D 0C 00 01 49  03 00 17 41 04 38 C0 4F   ...M...I ...A.8ÀO
 |                                                                                                0D 6B 50 23 9E 12 50 37  29 6F AD 4D 4A 8E EB D9   .kP#..P7 )o­MJ.ëÙ
 |                                                                                                5A 58 F8 C7 AE 32 9E 03  0B A3 03 51 51 F2 B7 39   ZXøÇ®2.. .£.QQò·9
 |                                                                                                52 A0 F1 73 D6 2D 33 E9  60 1A 48 6F D2 E6 02 57   R ñsÖ-3é `.HoÒæ.W
 |                                                                                                D9 58 F0 73 E1 5F 23 5C  0B CD 9E 06 DC 04 01 01   ÙXðsá_#\ .�..Ü...
 |                                                                                                00 B0 27 52 11 A6 5E 7A  8D 55 EE 6E AE 0C 7D D9   .°'R.¦^z .Uîn®.}Ù
 |                                                                                                09 FF 64 A1 8B 27 23 EE  BA 9F 48 0B 3B 71 A1 82   .ÿd¡.'#î º.H.;q¡.
 |                                                                                                BE 5A C2 2A 2B 25 60 84  40 B7 E3 39 C1 C0 73 94   ¾ZÂ*+%`. @·ã9�Às.
 |                                                                                                E8 8A 71 FD C2 07 41 5D  50 5F 66 6D 17 CB 03 66   è.qýÂ.A] P_fm.Ë.f
 |                                                                                                56 68 66 7E DE 14 0F 3A  BA 23 CA 81 47 6C B8 AB   Vhf~Þ..: º#Ê.Gl¸«
 |                                                                                                28 9A 2D 38 35 E5 D7 BC  A4 AF ED D2 C5 7F 7E 2E   (.-85å×¼ ¤¯íÒÅ.~.
 |                                                                                                9B FB 5D 6B F7 CB 6A 3E  C4 23 E5 12 04 7F B7 90   .û]k÷Ëj> Ä#å...·.
 |                                                                                                95 6A 4B D7 F6 AE E1 11  FB 1D BE 1F 1E 73 97 1B   .jK×ö®á. û.¾..s..
 |                                                                                                02 91 6D 8F EB F8 11 56  4F 0A 3D 7C AC DF CC B3   ..m.ëø.V O.=|¬ß̳
 |                                                                                                1E B6 85 0E 2E D4 DB BA  06 33 BA 41 76 A5 C8 85   .¶...ÔÛº .3ºAv¥È.
 |                                                                                                A5 06 6E 76 D4 61 9F 66  4F 0D 56 4A 93 F2 09 4A   ¥.nvÔa.f O.VJ.ò.J
 |                                                                                                86 D4 9F AB 3A 50 52 59  BF 5B 25 9F 7C F9 79 8C   .Ô.«:PRY ¿[%.|ùy.
 |                                                                                                5A FD F8 39 8E 99 CB 59  B9 2E 26 EA B3 98 86 D4   Zýø9..ËY ¹.&ê³..Ô
 |                                                                                                E8 1C A7 DF 98 8F 11 2B  A2 14 C3 6C 35 E6 8E 74   è.§ß...+ ¢.Ãl5æ.t
 |                                                                                                30 D9 50 B1 8F A2 80 ED  5F 70 6C 3F 0E F7 27 59   0ÙP±.¢.í _pl?.÷'Y
 |                                                                                                6D E6 29 BF 0E DD 01 4C  93 92 96 E1 1B F0 93 92   mæ)¿.�.L ...á.ð..
 |                                                                                                8F 16 03 03 00 04 0E 00  00 00                     ........ ..
 | [20.479s] Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware Debug: Authentication of the HTTPS connection timed out.
 | [20.535s] Microsoft.AspNetCore.Server.Kestrel.Connections Debug: Connection id "0HMAB2I8425HF" stopped.
 | [21.117s] Microsoft.Extensions.Hosting.Internal.Host Debug: Hosting stopping
 | [21.233s] Microsoft.Hosting.Lifetime Information: Application is shutting down...
 | [21.943s] Microsoft.Extensions.Hosting.Internal.Host Debug: Hosting stopped
 | [22.080s] Microsoft.AspNetCore.Server.Kestrel.InMemory.FunctionalTests.LoggingConnectionMiddlewareTests Error: Test threw an exception.
 | System.IO.IOException:  Received an unexpected EOF or 0 bytes from the transport stream.
 |    at System.Net.Security.SslStream.<FillHandshakeBufferAsync>g__InternalFillHandshakeBufferAsync|187_0[TIOAdapter](TIOAdapter adap, ValueTask`1 task, Int32 minSize) in System.Net.Security.dll:token 0x600030d+0x8b
 |    at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter) in System.Net.Security.dll:token 0x60002f1+0x8c
 |    at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) in System.Net.Security.dll:token 0x60002f0+0x299
 |    at Microsoft.AspNetCore.Server.Kestrel.InMemory.FunctionalTests.TestTransport.InMemoryHttpClientSlim.GetStream(Stream rawStream, Uri requestUri, Boolean validateCertificate) in /_/src/Servers/Kestrel/test/InMemory.FunctionalTests/TestTransport/InMemoryHttpClientSlim.cs:line 127
 |    at Microsoft.AspNetCore.Server.Kestrel.InMemory.FunctionalTests.TestTransport.InMemoryHttpClientSlim.GetStringAsync(Uri requestUri, Boolean validateCertificate) in /_/src/Servers/Kestrel/test/InMemory.FunctionalTests/TestTransport/InMemoryHttpClientSlim.cs:line 35
 |    at Microsoft.AspNetCore.Server.Kestrel.InMemory.FunctionalTests.TestTransport.InMemoryHttpClientSlim.GetStringAsync(String requestUri, Boolean validateCertificate) in /_/src/Servers/Kestrel/test/InMemory.FunctionalTests/TestTransport/InMemoryHttpClientSlim.cs:line 30
 |    at Microsoft.AspNetCore.Testing.TaskExtensions.TimeoutAfter[T](Task`1 task, TimeSpan timeout, String filePath, Int32 lineNumber) in /_/src/Shared/TaskExtensions.cs:line 90
 |    at Microsoft.AspNetCore.Server.Kestrel.InMemory.FunctionalTests.LoggingConnectionMiddlewareTests.LoggingConnectionMiddlewareCanBeAddedBeforeAndAfterHttps() in /_/src/Servers/Kestrel/test/InMemory.FunctionalTests/LoggingConnectionMiddlewareTests.cs:line 33
 |    at Microsoft.AspNetCore.Server.Kestrel.InMemory.FunctionalTests.LoggingConnectionMiddlewareTests.LoggingConnectionMiddlewareCanBeAddedBeforeAndAfterHttps() in /_/src/Servers/Kestrel/test/InMemory.FunctionalTests/LoggingConnectionMiddlewareTests.cs:line 35
 |    at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass48_1.<<InvokeTestMethodAsync>b__1>d.MoveNext() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 264
 | --- End of stack trace from previous location ---
 |    at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func`1 asyncAction) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\ExecutionTimer.cs:line 48
 |    at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in C:\Dev\xunit\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs:line 96
 | [24.078s] TestLifetime Information: Finished test LoggingConnectionMiddlewareCanBeAddedBeforeAndAfterHttps in 23.1967983s

Build

dougbu added a commit that referenced this issue Jul 20, 2021
- un-quarantine `ContentLength_Received_MultipleDataFramesOverSize_Reset`, #33373 fix seems to have worked
- quarantine `LoggingConnectionMiddlewareCanBeAddedBeforeAndAfterHttps()`, #34561
- skip `LongPollingTransportOnReceiveGetsCalled()` because we can't quarantine Java tests, #34563
@halter73
Copy link
Member

halter73 commented Jul 21, 2021

| [6.375s] Microsoft.AspNetCore.Server.Kestrel.Connections Debug: Connection id "0HMAB2I8425HF" accepted.
| [10.059s] Microsoft.AspNetCore.Server.Kestrel.Connections Debug: Connection id "0HMAB2I8425HF" started.

[20.479s] Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware Debug: Authentication of the HTTPS connection timed out.

It looks like there might have been some serious threadpool starvation for the "accepted" and "started" logs to be ~4 seconds apart. The only thing between those two logs is a threadpool dispatch. I think that's what caused the handshake timeout and failed test.

"Accepted" log:

Log.ConnectionAccepted(connection.ConnectionId);
KestrelEventSource.Log.ConnectionQueuedStart(connection);
ThreadPool.UnsafeQueueUserWorkItem(kestrelConnection, preferLocal: false);

Which immediately dispatches to the "started" log:

void IThreadPoolWorkItem.Execute()
{
_ = ExecuteAsync();
}
internal async Task ExecuteAsync()
{
var connectionContext = _transportConnection;
try
{
KestrelEventSource.Log.ConnectionQueuedStop(connectionContext);
Logger.ConnectionStart(connectionContext.ConnectionId);
KestrelEventSource.Log.ConnectionStart(connectionContext);

dougbu added a commit that referenced this issue Jul 21, 2021
- un-quarantine `ContentLength_Received_MultipleDataFramesOverSize_Reset`, #33373 fix seems to have worked
- quarantine `LoggingConnectionMiddlewareCanBeAddedBeforeAndAfterHttps()`, #34561
- skip `LongPollingTransportOnReceiveGetsCalled()` because we can't quarantine Java tests, #34563
@BrennanConroy BrennanConroy added this to the Next sprint planning milestone Jul 21, 2021
@ghost
Copy link

ghost commented Jul 21, 2021

Thanks for contacting us.

We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. We would like to keep this around to collect more feedback, which can help us with prioritizing this work. We will re-evaluate this issue, during our next planning meeting(s).
If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues.
To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

@HaoK HaoK self-assigned this Oct 14, 2021
@HaoK
Copy link
Member

HaoK commented Oct 14, 2021

Per discussion with @adityamandaleeka we are going to unquarantine this since its been well behaved and there isn't much actionable as the failure looks due to external issues like threadpool starvation

@HaoK
Copy link
Member

HaoK commented Oct 15, 2021

#37572

@HaoK HaoK closed this as completed Oct 15, 2021
@ghost ghost locked as resolved and limited conversation to collaborators Nov 14, 2021
@amcasey amcasey added area-networking Includes servers, yarp, json patch, bedrock, websockets, http client factory, and http abstractions and removed area-runtime labels Aug 24, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-networking Includes servers, yarp, json patch, bedrock, websockets, http client factory, and http abstractions test-failure
Projects
None yet
Development

No branches or pull requests

5 participants