HTTP2: Expect: 100-Continue support and early response handling

[geoffkizer]

We need to add logic to the HTTP implementation to support Expect: 100-continue.

In particular:
(1) If the user sets this header, defer sending the request body until we receive a either a 100 continue, or the 100-continue timeout expires
(2) If instead we receive a final response, and the status code is >=300, we should reset the request stream and stop sending it (if we started due to the timeout expiring)

We should also consider doing (2) above even without Expect: 100-continue. We don't do this for HTTP/1.1 currently, but it's not clear to me why we don't.

[geoffkizer]

For reference see the logic related to "hasExpectContinueHeader" in HttpConnection.SendAsyncCore.

[gmazzotti]

Hi,

I'm trying to implement an API integration with WebHDFS and I'm finding some problems with 100-continue thing.

Here is the api: https://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Create_and_Write_to_a_File

In the step 1, I used a try/catch block to handle the WebException when the server returns 307 for the redirection with the AllowAutoRedirect set to false.
However, in the Step 2, the WebHDFS server always return 100-continue before the final answer, even if, I set ServicePoint Expect100coninue to false.

So, we need to send the request with the Expect100continue handle process. I try to use HttpWebRequest and HttpClient classes, with the follow issues:

• HttpWebRequest throws WebException on 100-continue answer and gets the final status in the
  exception response body:

WebException message:

The remote server returned an error: (100) Continue.
Stack trace: at System.Net.HttpWebRequest.GetResponse()
at com.mztsolutions.WReqProcess.ExecuteSpecific() in C:\Users\GustavoMazzotti\Documents\workspace_vsts\MZT Solutions - 00 - Standard\com.mztsolutions\WReqProcess.cs:line 101

WebException – Response – Status = continue, body:

HTTP/1.1 201 Created
Location: hdfs://mztsandbox-hdp.mztsolutions.com:8020/user/appmanager/external/file01.txt
Content-Length: 0
Connection: close

• When I use HttpClient PutAsync method, the status of the response is 100-contniue and the body as the same as HttpWebRequest

Indeed, the HDFS writes the file but, I need to handle the response 201 by my own.

However, when error occurs on the server, for example, file exists, the server returns 403 and an error massage on the body:

In this case, the HttpWebRequest throws at 100-continue, but it is not possible to read the response body with the error:

The very same behavior is found in the HttpClient PutAsync method is used.

So, the .net standard classes fails completely to get the final answer when Expect 100-Continue is required.

In my case, I need to make a ASP.NET Core site writes uploaded files in the HDFS so, I cannot move to .NET Freamework that works right.

For now, the issue dotnet/dotnet-api-docs#1269 is not only a case of documentation, the classes are failing in that case.

As stated in https://github.com/dotnet/corefx/issues/23422#issuecomment-327589873, we can handle with the WebExceptions with try/catch, but in the case of 100-continue, we cannot.

But, in my opinion, WebException should only be used to 4xx status and few others. The 307 status of the step 1 of this API, for example, it is not a exception, it is a normal server response.

[karelz]

@gmazzotti your post seems to be unrelated to this issue. This is about HTTP/2 support of 100-Continue. HTTP/2 is now being added, it is not something that exists.
I will hide your post as off-topic. Please feel free to file a new issue with description of a problem you have - simple repro and clear description what is wrong, or what you want to add would be great. Thanks!

[karelz]

@wfurt are you actively working on it? (it is assigned to you)

[Tratcher]

(2) If instead we receive a final response, and the status code is >=300, we should reset the request stream and stop sending it (if we started due to the timeout expiring)

We should also consider doing (2) above even without Expect: 100-continue. We don't do this for HTTP/1.1 currently, but it's not clear to me why we don't.

Because for HTTP/1.1 that would close the socket and could lead to exhaustion. There's also a trade off for how long it takes to drain the request body vs how long it would take to establish a new socket connection (especially over TLS, proxies, etc). Using a request body drain timeout similar to the one for the response body might be a good compromise.

[Tratcher]

Note HttpWebRequest on 4.x has a mitigation for HTTP/1.1 chunked requests where it will send the chunked terminator instead of sending the request body for early 300+ response statuses. It also takes into account if the request body is buffered/re-transmittable, if not it will close the connection so that it can send the body only on the new request instead (e.g. auth, redirects, etc.).

[wfurt]

I was talking about some of the behavior with @mconnew in context of #26662. However, the current layering makes it somewhat challenging .

[Tratcher]

Ah, yeah, WCF was the customer for the HttpWebRequest change as well.