GRPC connecting over TOR

[JimmyMow]

Problem description

Hi. I am trying to connect to a server over Tor via grpc and haven't had any success. Is this because Tor has http/2 disabled for now? Has anyone had any success with grpc + Tor? Any pointers would be super helpful.

Reproduction steps

Attempt to connect to a server over Tor

[murgatroid99]

Can you be more specific about how your attempts to use gRPC with Tor have failed so far?

[mrfelton]

Hi @murgatroid99 . I've been running some tests on this:

I have a grpc service running on port 10009 and have a Tor set up to expose this endpoint.

If I try connecting over the .onion address with grpc-js I get a Failed to connect before the deadline timeout (connecting over the clearnet IP address directly works as expected).

I tried switching out grpc-js for grpc and tried the same again with debugging enabled. As you can see, it seems to be failing to resolve the .onion domain name:

I1009 02:43:10.001460000 4480984512 secure_channel_create.cc:201]      grpc_secure_channel_create(creds=0x10280dcb0, target=snhv55vqujlctlzwv4gavwzjlemafnbo7hexqcalh37iniolcspvfhad.onion:10009, args=0x102a42810, reserved=0x0)
I1009 02:43:10.001540000 4480984512 init.cc:164]                       grpc_init(void)
I1009 02:43:10.001561000 4480984512 client_idle_filter.cc:294]         (client idle filter) created with max_leisure_time = 1800000 ms
I1009 02:43:10.001574000 4480984512 client_channel.cc:1367]            chand=0x10481d860: creating client_channel for channel stack 0x10481d680
I1009 02:43:10.001975000 4480984512 channel_connectivity.cc:43]        grpc_channel_check_connectivity_state(channel=0x10481d600, try_to_connect=1)
I1009 02:43:10.001989000 4480984512 connectivity_state.cc:79]          CONWATCH: 0x10481d930 client_channel: get IDLE
I1009 02:43:10.002024000 4480984512 resolving_lb_policy.cc:196]        resolving_lb=0x102815450: starting name resolution
I1009 02:43:10.002030000 4480984512 client_channel.cc:1273]            chand=0x10481d860: update: state=CONNECTING picker=0x10281e880
I1009 02:43:10.002033000 4480984512 connectivity_state.cc:147]         SET: 0x10481d930 client_channel: IDLE --> CONNECTING [helper]
D1009 02:43:10.002039000 4480984512 dns_resolver.cc:242]               Start resolving.
I1009 02:43:10.002059000 4480984512 client_channel.cc:1551]            chand=0x10481d860: created resolving_lb_policy=0x102815450
I1009 02:43:10.002086000 4480984512 channel_connectivity.cc:232]       grpc_channel_watch_connectivity_state(channel=0x10481d600, last_observed_state=0, deadline=gpr_timespec { tv_sec: 1570581800, tv_nsec: 1000000, clock_type: 1 }, cq=0x102a3db60, tag=0x1028040a0)
I1009 02:43:10.002094000 4480984512 connectivity_state.cc:100]         CONWATCH: 0x10481d930 client_channel: from IDLE [cur=CONNECTING] notify=0x10280f8a0
I1009 02:43:10.002104000 4480984512 completion_queue.cc:682]           cq_end_op_for_next(cq=0x102a3db60, tag=0x1028040a0, error="No Error", done=0x105063480, done_arg=0x102811030, storage=0x102811130)
I1009 02:43:10.002127000 4480984512 completion_queue.cc:960]           grpc_completion_queue_next(cq=0x102a3db60, deadline=gpr_timespec { tv_sec: -9223372036854775808, tv_nsec: 0, clock_type: 0 }, reserved=0x0)
I1009 02:43:10.002135000 4480984512 completion_queue.cc:1060]          RETURN_EVENT[0x102a3db60]: OP_COMPLETE: tag:0x1028040a0 OK
I1009 02:43:10.002154000 4480984512 channel_connectivity.cc:43]        grpc_channel_check_connectivity_state(channel=0x10481d600, try_to_connect=1)
I1009 02:43:10.002206000 4480984512 connectivity_state.cc:79]          CONWATCH: 0x10481d930 client_channel: get CONNECTING
I1009 02:43:10.002230000 4480984512 channel_connectivity.cc:232]       grpc_channel_watch_connectivity_state(channel=0x10481d600, last_observed_state=1, deadline=gpr_timespec { tv_sec: 1570581800, tv_nsec: 1000000, clock_type: 1 }, cq=0x102a3db60, tag=0x102a42d30)
I1009 02:43:10.002240000 4480984512 connectivity_state.cc:100]         CONWATCH: 0x10481d930 client_channel: from CONNECTING [cur=CONNECTING] notify=0x102a36720
I1009 02:43:10.002270000 4480984512 completion_queue.cc:960]           grpc_completion_queue_next(cq=0x102a3db60, deadline=gpr_timespec { tv_sec: -9223372036854775808, tv_nsec: 0, clock_type: 0 }, reserved=0x0)
I1009 02:43:10.002280000 4480984512 completion_queue.cc:1060]          RETURN_EVENT[0x102a3db60]: QUEUE_TIMEOUT
I1009 02:43:10.003424000 4480984512 dns_resolver.cc:184]               dns resolution failed (will retry): {"created":"@1570581790.003407000","description":"getaddrinfo failed","file":"../deps/grpc/src/core/lib/iomgr/tcp_uv.cc","file_line":72,"grpc_status":14,"os_error":"unknown node or service"}
I1009 02:43:10.003475000 4480984512 resolving_lb_policy.cc:255]        resolving_lb=0x102815450: resolver transient failure: {"created":"@1570581790.003448000","description":"DNS resolution failed","file":"../deps/grpc/src/core/ext/filters/client_channel/resolver/dns/native/dns_resolver.cc","file_line":189,"grpc_status":14,"referenced_errors":[{"created":"@1570581790.003407000","description":"getaddrinfo failed","file":"../deps/grpc/src/core/lib/iomgr/tcp_uv.cc","file_line":72,"grpc_status":14,"os_error":"unknown node or service"}]}
I1009 02:43:10.003480000 4480984512 client_channel.cc:1273]            chand=0x10481d860: update: state=TRANSIENT_FAILURE picker=0x102a3ba60
I1009 02:43:10.003484000 4480984512 connectivity_state.cc:147]         SET: 0x10481d930 client_channel: CONNECTING --> TRANSIENT_FAILURE [helper]
I1009 02:43:10.003600000 4480984512 connectivity_state.cc:160]         NOTIFY: 0x10481d930 client_channel: 0x102a36720
D1009 02:43:10.003616000 4480984512 dns_resolver.cc:201]               retrying in 1000 milliseconds
I1009 02:43:10.003627000 4480984512 completion_queue.cc:682]           cq_end_op_for_next(cq=0x102a3db60, tag=0x102a42d30, error="No Error", done=0x105063480, done_arg=0x102a18760, storage=0x102a18860)
I1009 02:43:10.003638000 4480984512 completion_queue.cc:960]           grpc_completion_queue_next(cq=0x102a3db60, deadline=gpr_timespec { tv_sec: -9223372036854775808, tv_nsec: 0, clock_type: 0 }, reserved=0x0)
I1009 02:43:10.003644000 4480984512 completion_queue.cc:1060]          RETURN_EVENT[0x102a3db60]: OP_COMPLETE: tag:0x102a42d30 OK
I1009 02:43:10.003659000 4480984512 channel_connectivity.cc:43]        grpc_channel_check_connectivity_state(channel=0x10481d600, try_to_connect=1)
I1009 02:43:10.003665000 4480984512 connectivity_state.cc:79]          CONWATCH: 0x10481d930 client_channel: get TRANSIENT_FAILURE
I1009 02:43:10.003695000 4480984512 channel_connectivity.cc:232]       grpc_channel_watch_connectivity_state(channel=0x10481d600, last_observed_state=3, deadline=gpr_timespec { tv_sec: 1570581800, tv_nsec: 1000000, clock_type: 1 }, cq=0x102a3db60, tag=0x102a3d330)
I1009 02:43:10.003707000 4480984512 connectivity_state.cc:100]         CONWATCH: 0x10481d930 client_channel: from TRANSIENT_FAILURE [cur=TRANSIENT_FAILURE] notify=0x102a36720
I1009 02:43:10.003716000 4480984512 completion_queue.cc:960]           grpc_completion_queue_next(cq=0x102a3db60, deadline=gpr_timespec { tv_sec: -9223372036854775808, tv_nsec: 0, clock_type: 0 }, reserved=0x0)
I1009 02:43:10.003723000 4480984512 completion_queue.cc:1060]          RETURN_EVENT[0x102a3db60]: QUEUE_TIMEOUT
D1009 02:43:11.008246000 4480984512 dns_resolver.cc:242]               Start resolving.
I1009 02:43:11.008320000 4480984512 completion_queue.cc:960]           grpc_completion_queue_next(cq=0x102a3db60, deadline=gpr_timespec { tv_sec: -9223372036854775808, tv_nsec: 0, clock_type: 0 }, reserved=0x0)
I1009 02:43:11.008339000 4480984512 completion_queue.cc:1060]          RETURN_EVENT[0x102a3db60]: QUEUE_TIMEOUT
I1009 02:43:11.008357000 4480984512 completion_queue.cc:960]           grpc_completion_queue_next(cq=0x102a3db60, deadline=gpr_timespec { tv_sec: -9223372036854775808, tv_nsec: 0, clock_type: 0 }, reserved=0x0)
I1009 02:43:11.008368000 4480984512 completion_queue.cc:1060]          RETURN_EVENT[0x102a3db60]: QUEUE_TIMEOUT
I1009 02:43:11.011218000 4480984512 dns_resolver.cc:184]               dns resolution failed (will retry): {"created":"@1570581791.011140000","description":"getaddrinfo failed","file":"../deps/grpc/src/core/lib/iomgr/tcp_uv.cc","file_line":72,"grpc_status":14,"os_error":"unknown node or service"}

[murgatroid99]

If the addresses can't be resolved by our existing name resolution then it's not going to work with the grpc libraries as they currently exist. We don't have an API for resolving other kinds of names

[mrfelton]

According to https://2019.www.torproject.org/docs/tor-manual.html.en Tor now supports HTTPTunnelPort, i.e. serving as an http proxy.

HTTPTunnelPort [address:]port|auto [isolation flags]

Open this port to listen for proxy connections using the "HTTP CONNECT" protocol instead of SOCKS. Set this to 0 if you don’t want to allow "HTTP CONNECT" connections. Set the port to "auto" to have Tor pick a port for you. This directive can be specified multiple times to bind to multiple addresses/ports. If multiple entries of this option are present in your configuration file, Tor will perform stream isolation between listeners by default. See SOCKSPort for an explanation of isolation flags. (Default: 0)

I have successfully been able to use use gRPC over Tor by setting up a local Tor server that serves as an HTTP proxy using the HTTPTunnelPort option, and then routing the gRPC traffic through that by setting process.env.http_proxy.

This works with the native grpc library, which is able to resolve the .onion addresses.

However, it does not work with the grpc-js library which fails at the name resolution stage.

[murgatroid99]

The latest version of grpc-js now supports using HTTP CONNECT configured by the http_proxy environment variable. Does that work for you now?

[mrfelton]

Thats good progress @murgatroid99 . However, with grpc-js it's still not possible to connect to gRPC services over tor like you can with the native grpc because the js client is unable to resolve the .onion domains.

[murgatroid99]

Your previous trace log shows that the native grpc library also cannot resolve .onion addresses.

[mrfelton]

The earlier example was when trying to connect directly to a .onion address which doesn't work.

However it can be made to work by connecting via a local tor http tunnel.

Here is a full trace of a successful connection to .onion address via local tor tunnel using grpc native

https://gist.github.com/mrfelton/89fa4e9c0cff167ff949cba4c84081a5

[mrfelton]

FYI, you can reproduce this with LN-Zap/node-lnd-grpc#166

You can run this single test case to see it run.

GRPC_TRACE=all GRPC_VERBOSITY=DEBUG DEBUG=lnrpc* npx babel-tape-runner test/servives.Lightning.test.js | npx tap-spec

If you then swap out grpc for @grpc/grpc-js you can see how it fails in comparison.

I should also point out though that my test cases will fail when using grpc-js regarless due to #1354 - however I patched my local copy of grpc-js to fix the http_proxy env var support but it still fails at the name resolution phase.

[murgatroid99]

In any case, @grpc/grpc-js is just using dns.lookup under the hood, and we're not going to use dns.resolve* because it can't resolve localhost. So that part probably won't change.

[mrfelton]

Does it really need to do a dns lookup on the destination address when using a proxy?

[murgatroid99]

You're right. I published some new proxy handling code in grpc-js version 0.8.1, and it doesn't actually do that DNS call. That might work for you.

[mrfelton]

I'm getting a different result with 0.8.1 - definitely looks like an improvement, though it still appears to have an issue.

See logs here https://gist.github.com/mrfelton/50a517b223af0e4846e87aa24b8ba6b3

The logs indicate that the proxy was indeed used (so #1354 no longer seems to be an issue), and that a connection was established. However, the actual grpc call is hanging indefinitely at the last line. I'm expecting to receive an UNAVAILABLE status back from this endpoint but seems I'm getting no response.

[murgatroid99]

Can you get logs from the proxy server, to verify that data is getting sent through the HTTP CONNECT tunnel? That client definitely thinks it's sending the request to the server.