network interface binding option

[crucifyer]

hi,

I wish bind address option.
thank you.

ex) pc ips (1.1.1.2, 1.1.1.3, 1.1.1.4)

`# ip a

2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether ****
inet 1.1.1.2/24 brd 1.1.1.255 scope global enp2s0
inet 1.1.1.3/24 brd 1.1.1.255 scope global enp2s0
inet 1.1.1.4/24 brd 1.1.1.255 scope global enp2s0`

I wish choice 1.1.1.3

`import http.client

conn = http.client.HTTPConnection('xenosi.de', source_address=tuple(['1.1.1.3', 0]));

h = {}
h['User-Agent'] = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36'
conn.request('GET', '/ip.php?json', headers=h);

res = conn.getresponse();
print(res.status, res.reason, res.read())`

result)
proxy --hostname=0.0.0.0 --bindip=1.1.1.3

[crucifyer]

proxy --hostname=1.1.1.3 is not solution. proxy using 1.1.1.2

[abhinavsingh]

Hi @crucifyer , IMO using desired interface IP with --hostname flag must do the trick.

Can you post the logs printed, when you start proxy with custom --hostname flag? Thank you!!!

[crucifyer]

# proxy --hostname=1.1.1.3
2021-04-08 16:15:01,771 - pid:1980 [I] load_plugins:334 - Loaded plugin proxy.http.proxy.HttpProxyPlugin
2021-04-08 16:15:01,772 - pid:1980 [I] listen:115 - Listening on 1.1.1.3:8899
2021-04-08 16:15:01,777 - pid:1980 [I] start_workers:136 - Started 2 workers
2021-04-08 16:15:20,754 - pid:1983 [I] access_log:397 - 3.4.5.6:59064 - CONNECT ogs.google.com:443 - 17772 bytes - 10508.13 ms
2021-04-08 16:15:21,073 - pid:1984 [I] access_log:397 - 3.4.5.6:59067 - CONNECT www.gstatic.com:443 - 792 bytes - 10320.72 ms
2021-04-08 16:15:21,090 - pid:1983 [I] access_log:397 - 3.4.5.6:59066 - CONNECT www.google.com:443 - 4121 bytes - 10447.67 ms
2021-04-08 16:15:21,255 - pid:1983 [I] access_log:397 - 3.4.5.6:59068 - CONNECT apis.google.com:443 - 1531 bytes - 10499.93 ms
2021-04-08 16:15:21,378 - pid:1984 [I] access_log:397 - 3.4.5.6:59069 - CONNECT lh3.google.com:443 - 2865 bytes - 10576.36 ms
2021-04-08 16:15:21,925 - pid:1983 [I] access_log:397 - 3.4.5.6:59070 - CONNECT lh3.googleusercontent.com:443 - 17078 bytes - 10553.79 ms
2021-04-08 16:15:22,163 - pid:1984 [I] access_log:397 - 3.4.5.6:59071 - CONNECT xenosi.de:443 - 1828 bytes - 10266.32 ms
2021-04-08 16:15:22,452 - pid:1984 [I] access_log:397 - 3.4.5.6:59065 - CONNECT ssl.gstatic.com:443 - 1566 bytes - 12104.30 ms```


but, 3.4.5.6 browser connect proxy https://xenosi.de/ip.php return 1.1.1.2
thank you.

[abhinavsingh]

I am unsure what that ip.php file is. If it simply returns IP address (like httpbin.org/get) then 1.1.1.2 is most likely your public facing interface, and hence you receive 1.1.1.2 in response. proxy.py itself is listening and serving over 1.1.1.3. You are already sending traffic to 1.1.1.3 (from browser) to access the proxy, right?

On Thu, 8 Apr 2021 at 12:49 PM, Song Hyo Jin ***@***.***> wrote: # proxy --hostname=1.1.1.3 2021-04-08 16:15:01,771 - pid:1980 [I] load_plugins:334 - Loaded plugin proxy.http.proxy.HttpProxyPlugin 2021-04-08 16:15:01,772 - pid:1980 [I] listen:115 - Listening on 1.1.1.3:8899 2021-04-08 16:15:01,777 - pid:1980 [I] start_workers:136 - Started 2 workers 2021-04-08 16:15:20,754 - pid:1983 [I] access_log:397 - 3.4.5.6:59064 - CONNECT ogs.google.com:443 - 17772 bytes - 10508.13 ms 2021-04-08 16:15:21,073 - pid:1984 [I] access_log:397 - 3.4.5.6:59067 - CONNECT www.gstatic.com:443 - 792 bytes - 10320.72 ms 2021-04-08 16:15:21,090 - pid:1983 [I] access_log:397 - 3.4.5.6:59066 - CONNECT www.google.com:443 - 4121 bytes - 10447.67 ms 2021-04-08 16:15:21,255 - pid:1983 [I] access_log:397 - 3.4.5.6:59068 - CONNECT apis.google.com:443 - 1531 bytes - 10499.93 ms 2021-04-08 16:15:21,378 - pid:1984 [I] access_log:397 - 3.4.5.6:59069 - CONNECT lh3.google.com:443 - 2865 bytes - 10576.36 ms 2021-04-08 16:15:21,925 - pid:1983 [I] access_log:397 - 3.4.5.6:59070 - CONNECT lh3.googleusercontent.com:443 - 17078 bytes - 10553.79 ms 2021-04-08 16:15:22,163 - pid:1984 [I] access_log:397 - 3.4.5.6:59071 - CONNECT xenosi.de:443 - 1828 bytes - 10266.32 ms 2021-04-08 16:15:22,452 - pid:1984 [I] access_log:397 - 3.4.5.6:59065 - CONNECT ssl.gstatic.com:443 - 1566 bytes - 12104.30 ms``` but, 3.4.5.6 browser connect proxy https://xenosi.de/ip.php return 1.1.1.2 thank you. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#535 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA6Y4KMHX57OYTXLUQUVEDTHVKIRANCNFSM42SESC5Q> .

-- ------------------------------------------------------ Abhinav Singh, https://abhinavsingh.com ------------------------------------------------------

[crucifyer]

Of course.
The foxy proxy is set to 1.1.1.3:8899.

I have a lot of ips on my server.
I want to pick one of them and use it.
Add an option that accepts source_address= like the code above.

The exact purpose is
PCs in the office intranet do not have Internet by themselves,
This is to make the Internet work only by proxy settings in the browser.

thank you.

[crucifyer]

Maybe this is the part.
Wouldn't it be possible to apply it with setsockopt?

https://docs.python.org/3/library/socket.html

# create a raw socket and bind it to the public interface
s = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_IP)
s.bind((HOST, 0))

# Include IP headers
s.setsockopt(socket.IPPROTO_IP, socket.IP_HDRINCL, 1)

Please understand that this may be wrong as this is not well known.
thank you.

[abhinavsingh]

Maybe this is the part.
Wouldn't it be possible to apply it with setsockopt?

https://docs.python.org/3/library/socket.html

# create a raw socket and bind it to the public interface
s = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_IP)
s.bind((HOST, 0))

# Include IP headers
s.setsockopt(socket.IPPROTO_IP, socket.IP_HDRINCL, 1)

Please understand that this may be wrong as this is not well known.
thank you.

@crucifyer This possibly can work (haven't tried it myself before) but will definitely require sudo privileges.

Can you work on a small POC, where:

1. Simple take your code above
2. Establish a connection to upstream server
3. Check upstream logs to see if IP header indeed gets injected

If it does, we can wrap this POC as a feature within proxy.py, wdyt?

[crucifyer]

I tested it like this. The bind() alone works very well.

import sys

HOST = 'xenosi.de'
PORT = 80
s = None

for res in socket.getaddrinfo(HOST, PORT, socket.AF_UNSPEC, socket.SOCK_STREAM):
    af, socktype, proto, canonname, sa = res
    try:
        s = socket.socket(af, socktype, proto)
    except OSError as msg:
        s = None
        print(msg)
        continue
    try:
        s.bind(('1.1.1.3', 0))
        s.connect(sa)
    except OSError as msg:
        s.close()
        s = None
        print(msg)
        continue
    break

if s is None:
    print('could not open socket')
    sys.exit(1)

with s:
    s.sendall(b'GET /ip.php?json HTTP/1.1\r\nHOST: xenosi.de\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36\r\n\r\n')
    data = s.recv(1024)

print(data)

b'HTTP/1.1 200 OK\r\nDate: Sun, 11 Apr 2021 06:57:32 GMT\r\nContent-Type: application/x-javascript; charset=utf-8\r\n
Transfer-Encoding: chunked\r\n
Connection: keep-alive\r\n
Set-Cookie: __cfduid=da71asdf8ab352awef7e3eee0dd61618124252; expires=Tue, 11-May-21 06:57:32 GMT; path=/; domain=.xenosi.de; HttpOnly; SameSite=Lax\r\n
X-Powered-By: PHP/5.5.9-1ubuntu4.29\r\n
Strict-Transport-Security: max-age=63072000; preload\r\n
CF-Cache-Status: DYNAMIC\r\ncf-request-id: 096151bcb5000asdf8000000001\r\n
Report-To: {"max_age":604800,"endpoints":[{"url":"https:\\/\\/a.nel.cloudflare.com\\/report?s=esqZBasdf4KsVCy%2FkMT6Z5tPhdasdfH8t1JlxkIQ0YtlDLpcu5dLycGX3ZqQ%3D"}],"group":"cf-nel"}\r\nNEL: {"report_to":"cf-nel","max_age":604800}\r\nServer: cloudflare\r\nCF-RAY: 63e252412edf0a9c-NRT\r\nalt-svc: h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400\r\n\r\n15\r\n
{"ip":"1.1.1.3"}\r\n
0\r\n\r\n'

thank you.

[abhinavsingh]

Thank you @crucifyer , but I am left a little confused over what is our objective here. If I understand correctly, you did a bind on 1.1.1.3 and upstream server saw the request coming from 1.1.1.3.

I think same should be achievable using proxy --hostname 1.1.1.3. Is you upstream server seeing 1.1.1.2 when making a request from proxy.py? If yes, may be we must put the bind login you demonstrated. Please let me know.

[crucifyer]

yes.
I think, need bind on self.client

[crucifyer]

I want to put a bind statement, but I can't find where to make the client socket. 😿

[abhinavsingh]

@crucifyer You may hardcode bind here in the constructor of TcpClientConnection class https://github.com/abhinavsingh/proxy.py/blob/develop/proxy/core/connection/client.py#L25. If it works out, we can later add this in a more generic sense.

[abhinavsingh]

@crucifyer Are you still here with me on this :). I am looking into it but would like to clarify what exactly are we after. Your example is a little misaligned, because bind is supposed to be used by servers and connect by clients. But IIUC, you are trying to call bind and connect on the same socket object. This is not a valid usage.

May you be can convey your requirement via this diagram and let me know where exactly are you expecting bind.

More useful will be to explain the scenario you are trying to achieve, Thank you!!!

[crucifyer]

@abhinavsingh

What I need is this.
Bind is also required when creating a socket for external querying.
server 'listen socket' is not 'query socket'
thank you.

[abhinavsingh]

@crucifyer Ok, so I think we need to pass source_address parameter to underlying socket.create_connection to make this work.

proxy.py calls this from within new_socket_connection in utiils.py. See https://github.com/abhinavsingh/proxy.py/blob/develop/proxy/common/utils.py#L208

Can you try to pass source_address here and see if it achieves the desired result for you?

I am already working on a custom DNS resolver plugin, see #664 and I am thinking same resolution callback can also provide the source address to use. Allowing you to override source_address parameter via a plugin.

PTAL and Lmk. Thank you.

[abhinavsingh]

Please take a look at #671

This should do exactly what we are after here. If it doesn't let me know. I couldn't try it immediately for verification, as I'll need a host with 2 public IPs configured.

Please give it a try when you can. I added a CustomDnsResolver plugin which can be used to configure source_address too. See doc string for explanation.

Let me know if you run into any issue.

[crucifyer]

@abhinavsingh yes yes!! it works!
thank you!
return socket.create_connection(addr, timeout=timeout, source_address=("x.x.x.x",0))

[abhinavsingh]

@abhinavsingh yes yes!! it works! thank you! return socket.create_connection(addr, timeout=timeout, source_address=("x.x.x.x",0))

Awesome, if you want, you can now use a plugin to customize this behavior and need not be hardcoded.

Example, extend https://github.com/abhinavsingh/proxy.py/blob/develop/proxy/plugin/custom_dns_resolver.py#L32 plugin and return None, ("x.x.x.x", 0) from within resolve_dns callback to make this works.

Thank you for bringing this to my notice.

[abhinavsingh]

@crucifyer See some documentation here https://github.com/abhinavsingh/proxy.py#customdnsresolverplugin. Lemme know if you run into any issue again :)

[crucifyer]

@abhinavsingh
It works! Now do is replace it with the hostname variable!
return None, ("x.x.x.x", 0)

[abhinavsingh]

Awesome. I think you might be able to use one of these combination to achieve the necessary:

>>> socket.gethostbyaddr("127.0.0.1")
('localhost', ['1.0.0.127.in-addr.arpa'], ['127.0.0.1'])

>>> socket.gethostbyname("localhost")
'127.0.0.1'

Wdyt?

[crucifyer]

I meant args like this.
return None, (self.args.hostname, 0)

[abhinavsingh]

I meant args like this. return None, (self.args.hostname, 0)

Cool, I see what you mean here. Essentially, use the same interface as the one defined via --hostname parameter.

Good point. I think this should be the default behavior. And we should be able to bake it into the core, so that a plugin is no longer necessary for your use case.

A plugin will still be required if you want to have a different source address as that defined by --hostname flag.

Wdyt?

[saldiray06]

is there any way listen 2 ports and bind data to any one of external ip?
Ex: listen 1881 and 1882
1881 internal port to externals 192.168.10.100:80
1882 internal port to externals 192.168.11.100:80

[abhinavsingh]

v2.4.0 added --ports flag, which can be used to listen on multiple ports. --port flag is still required. See some info in README

Here is how to use it

1. proxy --ports 9000 : Will listen on 8899 (default) and 9000
2. proxy --port 1881 --ports 1882 : Will listen on 1881 and 1882

[abhinavsingh]

1881 internal port to externals 192.168.10.100:80
1882 internal port to externals 192.168.11.100:80

This should be possible, but unfortunately, you will also need to know on which port was the request accepted 1881 or 1882. Currently, this information is not exposed to the plugins :(. I have myself run into this issue before and I hope to get this addressed, now that we have someone else in need of this feature :). I have opened

• [Plugin] Expose listening port to plugins #1085

to track support for this.

[saldiray06]

Great thnx

[theol-git]

Here is a monkey-patch i added to be able to select an interface by name in the plugin:

import ipaddress
import socket
import sys

# Monkey patch to allow the selection of a network interface for the proxy to function
from proxy.common import utils


def patched_new_socket_connection(
        addr,
        timeout: float = 10.0,
        source_address=None,
) -> socket.socket:
    conn = None
    try:
        ip = ipaddress.ip_address(addr[0])
        if ip.version == 4:
            conn = socket.socket(
                socket.AF_INET,
                socket.SOCK_STREAM,
                0,
            )
            if source_address and "." not in source_address:
                conn.setsockopt(
                    socket.SOL_SOCKET, socket.SO_BINDTODEVICE, str(source_address + "\0").encode("utf-8")
                )
            conn.settimeout(timeout)
            conn.connect(addr)
        else:
            conn = socket.socket(
                socket.AF_INET6,
                socket.SOCK_STREAM,
                0,
            )
            if source_address and "." not in source_address:
                conn.setsockopt(
                    socket.SOL_SOCKET, socket.SO_BINDTODEVICE, str(source_address + "\0").encode("utf-8")
                )
            conn.settimeout(timeout)
            conn.connect((addr[0], addr[1], 0, 0))
    except ValueError:
        pass  # does not appear to be an IPv4 or IPv6 address

    if conn is not None:
        return conn
    
    # try to establish dual stack IPv4/IPv6 connection.
    return socket.create_connection(
        addr, timeout=timeout, source_address=source_address
    )


def uncache(exclude):
    """Remove package modules from cache except excluded ones.
    On next import they will be reloaded.

    Args:
        exclude (iter<str>): Sequence of module paths.
    """
    pkgs = []
    for mod in exclude:
        pkg = mod.split(".", 1)[0]
        pkgs.append(pkg)

    to_uncache = []
    for mod in sys.modules:
        if mod in exclude:
            continue

        if mod in pkgs:
            to_uncache.append(mod)
            continue

        for pkg in pkgs:
            if mod.startswith(pkg + "."):
                to_uncache.append(mod)
                break

    for mod in to_uncache:
        del sys.modules[mod]


utils.new_socket_connection = patched_new_socket_connection
uncache(["proxy.common.utils"])

Here is the code for the plugin:

try:
    return (
        socket.getaddrinfo(host, port, proto=socket.IPPROTO_TCP,)[0][
            4
        ][0],
        interface_name,
    )
except socket.gaierror:
    raise HttpProtocolException()

interface_name is a string of the interface's name (ex: eth1)