Proxy.py with Firefox on NetBSD

python proxy.py --port 12500 --ipv4

then configure Firefox proxy 127.0.0.1:12500

it gives errors and not working

 return recvfds(s, 1)[0]
  File "/usr/lib/python3.7/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata

[abhinavsingh]

Few questions:

a) What OS are you using?
b) What happens if you try to use proxy.py using curl, example: curl -v -x 127.0.0.1:12500 http://httpbin.org/get
c) I believe there is more stacktrace than what you shared, can you share the full trace.
d) I double verified locally (MacOS) and Firefox seems to be working fine for me. Wondering are you also sending SOCKS/FTP/SSL traffic to proxy.py? Firefox seems to also provide configuration for it. Here is how it looks like for me.

[abhinavsingh]

@mbatrawi Above I only configured Firefox for HTTP traffic. You should also fill in proxy details for SSL proxy to proxy HTTPS connect requests.

Enabling proxy.py for FTP and SOCKS proxy may result in undefined behavior.

curl -v -x 127.0.0.1:12500 http://httpbin.org/get
* Expire in 0 ms for 6 (transfer 0x6fdec033c000)
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x6fdec033c000)
* Connection failed
* connect to 127.0.0.1 port 12500 failed: Connection refused
* Failed to connect to 127.0.0.1 port 12500: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 127.0.0.1 port 12500: Connection refused

I am using
NetBSD NetBSD 8.1 (GENERIC) #0
machine amd64

I tried Both fill all for firefox and socks5 only but keep in your mind I am running proxy.py on one machine and firefox on different machine using this command

python3.6 proxy.py --port 12500 --ipv4 --hostname 0.0.0.0

and configure firefox IP:12500

[abhinavsingh]

Thanks for the info.

Running proxy.py and Firefox on separate machines should not matter, I expect things to work fine. I see you are also using --ipv4 --hostname 0.0.0.0 options, so proxy.py should be reachable from an external machine on the network.

However, I don't quite understand your curl output. Somehow, you are unable to reach proxy.py via curl.

1. Are you sure you had proxy.py started before running the curl command?
2. Were you running curl command on the same machine where proxy.py was running, if not try with machine IP.

If both 1) and 2) are OK, then its obvious there is something else prohibiting connections to proxy.py, may be a firewall?

Also, you should NOT setup Firefox to use proxy.py as SOCKS or FTP proxy.

1. corrected

curl -v -x 0.0.0.0:12500 http://httpbin.org/get

* Expire in 0 ms for 6 (transfer 0x7e349ed3c000)
*   Trying 0.0.0.0...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x7e349ed3c000)
* Connected to 0.0.0.0 (127.0.0.1) port 12500 (#0)
> GET http://httpbin.org/get HTTP/1.1
> Host: httpbin.org
> User-Agent: curl/7.64.0
> Accept: */*
> Proxy-Connection: Keep-Alive

2. I configured firefox as you mentioned above exactly moreover I tried to fill ssl ftp
   no difference in both cases and not working

$ curl -v -x 0.0.0.0:12500 http://httpbin.org/get

* Expire in 0 ms for 6 (transfer 0x7c7e6bb3c000)
*   Trying 0.0.0.0...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x7c7e6bb3c000)
* Connected to 0.0.0.0 (127.0.0.1) port 12500 (#0)
> GET http://httpbin.org/get HTTP/1.1
> Host: httpbin.org
> User-Agent: curl/7.64.0
> Accept: */*
> Proxy-Connection: Keep-Alive
>
* Empty reply from server
* Connection #0 to host 0.0.0.0 left intact
curl: (52) Empty reply from server

[abhinavsingh]

So now we have curl connecting to proxy.py, however you get an empty response. Frankly I haven't personally tried proxy.py on NetBSD yet, so we might be chasing some bug here. To help debug further can you try this:

1. Start proxy.py with debug level logging in Terminal 1

python3.6 proxy.py --port 12500 --ipv4 --hostname 0.0.0.0 --log-level d

2. On Terminal 2, execute the same curl command again

curl -v -x 0.0.0.0:12500 http://httpbin.org/get

Share the exception / stacktrace printed on Terminal 1.

[abhinavsingh]

@mbatrawi I said share "exception / stacktrace", infact share everything printed on Terminal 1 to help understand the flow better. Thank you!!!

[abhinavsingh]

Searched for RuntimeError: received 0 items of ancdata and found few similar issues in other Python packages. This comment suggests increasing ulimit on the system fixed their issue.

proxy.py automatically tries to set ulimit to 1024. If you start proxy.py with debug logging, you should see something like this:

$ ./proxy.py --log-level d
2019-09-15 20:09:22,894 - DEBUG - pid:45584 - set_open_file_limit:1132 - Open file descriptor soft limit set to 1024

Try increasing the ulimit to 4096 for your system and see if that helps. Pass following command line option too --open-file-limit 4096 when starting proxy.py

2019-09-16 04:17:53,993 - INFO - pid:6667 - load_plugins:1151 - Loaded plugin <class 'proxy.HttpProxyPlugin'>
2019-09-16 04:17:53,994 - INFO - pid:6667 - setup:262 - Starting 8 workers
2019-09-16 04:17:54,127 - INFO - pid:6667 - run:225 - Started server on 0.0.0.0:12500
2019-09-16 04:19:08,077 - DEBUG - pid:6667 - handle:276 - Dispatched client request to worker id 0
Process Worker-1:
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "proxy.py", line 309, in run
    op, payload = self.work_queue.get(True, 1)
  File "/usr/pkg/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 239, in _rebuild_socket
    fd = df.detach()
  File "/usr/pkg/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
2019-09-16 04:19:46,803 - DEBUG - pid:6667 - handle:276 - Dispatched client request to worker id 1
Process Worker-2:
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "proxy.py", line 309, in run
    op, payload = self.work_queue.get(True, 1)
  File "/usr/pkg/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 239, in _rebuild_socket
    fd = df.detach()
  File "/usr/pkg/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
2019-09-16 04:19:49,786 - DEBUG - pid:6667 - handle:276 - Dispatched client request to worker id 2
Process Worker-3:
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "proxy.py", line 309, in run
    op, payload = self.work_queue.get(True, 1)
  File "/usr/pkg/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 239, in _rebuild_socket
    fd = df.detach()
  File "/usr/pkg/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
2019-09-16 04:19:52,762 - DEBUG - pid:6667 - handle:276 - Dispatched client request to worker id 3
Process Worker-4:
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "proxy.py", line 309, in run
    op, payload = self.work_queue.get(True, 1)
  File "/usr/pkg/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 239, in _rebuild_socket
    fd = df.detach()
  File "/usr/pkg/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
2019-09-16 04:19:56,616 - DEBUG - pid:6667 - handle:276 - Dispatched client request to worker id 4
2019-09-16 04:19:56,616 - DEBUG - pid:6667 - handle:276 - Dispatched client request to worker id 5
Process Worker-5:
Process Worker-6:
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "proxy.py", line 309, in run
    op, payload = self.work_queue.get(True, 1)
  File "/usr/pkg/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 239, in _rebuild_socket
    fd = df.detach()
  File "/usr/pkg/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/pkg/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "proxy.py", line 309, in run
    op, payload = self.work_queue.get(True, 1)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/pkg/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 239, in _rebuild_socket
    fd = df.detach()
  File "/usr/pkg/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
RuntimeError: received 0 items of ancdata
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
2019-09-16 04:19:57,522 - DEBUG - pid:6667 - handle:276 - Dispatched client request to worker id 6
Process Worker-7:
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "proxy.py", line 309, in run
    op, payload = self.work_queue.get(True, 1)
  File "/usr/pkg/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 239, in _rebuild_socket
    fd = df.detach()
  File "/usr/pkg/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
2019-09-16 04:19:59,000 - DEBUG - pid:6667 - handle:276 - Dispatched client request to worker id 7
Process Worker-8:
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "proxy.py", line 309, in run
    op, payload = self.work_queue.get(True, 1)
  File "/usr/pkg/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 239, in _rebuild_socket
    fd = df.detach()
  File "/usr/pkg/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
2019-09-16 04:19:59,031 - DEBUG - pid:6667 - handle:276 - Dispatched client request to worker id 0
2019-09-16 04:19:59,854 - DEBUG - pid:6667 - handle:276 - Dispatched client request to worker id 1
2019-09-16 04:19:59,855 - DEBUG - pid:6667 - handle:276 - Dispatched client request to worker id 2
2019-09-16 04:20:07,497 - DEBUG - pid:6667 - handle:276 - Dispatched client request to worker id 3
2019-09-16 04:20:39,223 - DEBUG - pid:6667 - handle:276 - Dispatched client request to worker id 4