Inconsistent header transforms defaults between YARP and Forwarder

[dariusz-wozniak]

YARP Direct Proxy for one of the endpoint screams with the following error:

The SSL connection could not be established, see inner exception

Inner exception says:

Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.

In the current, non-YARP proxy solution, the problem is solved (or rather, by-passed) by setting the ServerCertificateCustomValidationCallback flag:

public class IgnoreSslHandler : HttpClientHandler
{
    public IgnoreSslHandler()
    {
        if (Environment.IsDevelopment())
        {
            ServerCertificateCustomValidationCallback = DangerousAcceptAnyServerCertificateValidator;
        }
    }
}

But, HttpMessageInvoker with the HttpClientHandler that has the ServerCertificateCustomValidationCallback flag set to either true or to DangerousAcceptAnyServerCertificateValidator doesn't seem to be used in the YARP.

Code snippet that is used for proxy context:

await httpProxy.ProxyAsync(httpContext, "https://url", httpClient, requestOptions, transformer);

Any idea how to solve that?

[karelz]

If you have exactly the same HttpClient config outside of YARP and in YARP, I would expect same results.
So, the key question is find out what is the difference. Either you can look at the HttpClient setup, or you can collect and analyze network traces to see what is happening and why is the connection closed by the server.

[dariusz-wozniak]

Thank you for the reply, @karelz.

The key difference is non-YARP solution uses HttpClient while YARP does not allow to use HttpClient directly; see part of the Yarp.ReverseProxy.Service.Proxy.HttpProxy.ProxyAsync below:

// HttpClient overload for SendAsync changes response behavior to fully buffered which impacts performance
// See discussion in https://github.com/microsoft/reverse-proxy/issues/458
if (httpClient is HttpClient)
{
    throw new ArgumentException($"The http client must be of type HttpMessageInvoker, not HttpClient", nameof(httpClient));
}

So, I'm using the HttpClientHandler in the same way as in the documentation, but with additional ServerCertificateCustomValidationCallback flag:

var httpMessageHandler = new HttpClientHandler()
{
    UseProxy = false,
    AllowAutoRedirect = false,
    AutomaticDecompression = DecompressionMethods.None,
    UseCookies = false,
    ServerCertificateCustomValidationCallback = (_, _, _, _) => true
};

var httpClient = new HttpMessageInvoker(httpMessageHandler);

And, when debugging the code, true of the ServerCertificateCustomValidationCallback property is never hit that implies this flag might be skipped when processing.

you can collect and analyze network traces to see what is happening and why is the connection closed by the server

What application do you recommend for that?

[Tratcher]

HttpClient vs HttpMessageInvoker shouldn't affect the SSL behavior.

Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.

That says the remote server terminated the connection, that's not caused by a trust issue on your end. Do you have access to the remote server logs to find out why it killed the connection?

[dariusz-wozniak]

Thank you for the answer, @Tratcher.

Unfortunately, I don't have an access to the server logs.

Some more information on the topic after more investigation:

• Endpoint does not have valid certificate. When sending a request via Postman with the "SSL certificate verification" turned on, I get the "SSL Error: Unable to get local issuer certificate" error message. Thus, when sending a request from the Postman, I need to disable that option.

• And, I'm sending the same request to the proxy. Direct request is successful while request via proxy is not.

• I tried to use HttpClient and skip the if-statement (if (httpClient is HttpClient) within the HttpProxy.cs) while debuggning to allow HttpClient to be used by the YARP, but got the error, even if flag ServerCertificateCustomValidationCallback is set to true. So, the below @Tratcher's assumption seems to be true:

HttpClient vs HttpMessageInvoker shouldn't affect the SSL behavior.

• The interesting fact is that, when sending request via Postman through the proxy, Fiddler shows 502 error that says:

[Fiddler] The connection to '[...]' failed.
Error: TimedOut (0x274c).
System.Net.Sockets.SocketException A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond [...]

The URL indicates the remote .crt location.

The process is either iisexpress or the project itself depending on how it is running.

• Currently, I'm using Refit as a reverse proxy. Refit uses HttpClient underneath and I'm using the IgnoreSslHandler from the very first post in that thread.

[Tratcher]

Ok, one more theory: HttpTransformer.TransformRequestAsync copies over all the request headers, including the external Host. That Host is used in the TLS SNI handshake rather than the host specified in the destination url. The destination doesn't have a certificate matching that external Host so it refuses the connection.

In the full proxy flow we remove the external host by default to avoid this kind of conflict. You can do this in the direct proxy like this:
https://github.com/microsoft/reverse-proxy/blob/2593b4c009f2c3a131cd4bf327115976aad2d7a6/samples/ReverseProxy.Direct.Sample/Startup.cs#L98-L99

HttpTransformer.Default should probably be updated to reflect the same defaults as the full proxy flow:

• X-Forwarders on
• OriginalHost off

[dariusz-wozniak]

@thatcher: Is there any way I can proof that my flow is working correctly with the settings you mentioned?

[Tratcher]

What part are you trying to prove? Is it still not working for you?

A network trace would be able to capture the TLS handshake (that part's not encrypted).

[dariusz-wozniak]

Yes, that is working now. Setting proxyRequest.Headers.Host to null resolves the issue. I just slightly misunderstood your answer on that.

Thank you a lot, @Tratcher.

[Tratcher]

Reopening. HttpTransformer.Default should probably be updated to reflect the same defaults as the full proxy flow:

• X-Forwarders on
• OriginalHost off

[karelz]

Triage:

• YARP removes original Host header and then defaults to Destination host
  • Forwarder just copies the header --> it should do the same
• X-Forwarders transforms in YARP are enabled, but not when Forwarder is used