Proxy.py with Firefox on NetBSD #68

ghost · 2019-09-15T16:07:30Z

python proxy.py --port 12500 --ipv4

then configure Firefox proxy 127.0.0.1:12500

it gives errors and not working

 return recvfds(s, 1)[0]
  File "/usr/lib/python3.7/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata

The text was updated successfully, but these errors were encountered:

abhinavsingh · 2019-09-16T01:23:27Z

Few questions:

a) What OS are you using?
b) What happens if you try to use proxy.py using curl, example: curl -v -x 127.0.0.1:12500 http://httpbin.org/get
c) I believe there is more stacktrace than what you shared, can you share the full trace.
d) I double verified locally (MacOS) and Firefox seems to be working fine for me. Wondering are you also sending SOCKS/FTP/SSL traffic to proxy.py? Firefox seems to also provide configuration for it. Here is how it looks like for me.

abhinavsingh · 2019-09-16T01:59:47Z

@mbatrawi Above I only configured Firefox for HTTP traffic. You should also fill in proxy details for SSL proxy to proxy HTTPS connect requests.

Enabling proxy.py for FTP and SOCKS proxy may result in undefined behavior.

ghost · 2019-09-16T02:01:33Z

curl -v -x 127.0.0.1:12500 http://httpbin.org/get
* Expire in 0 ms for 6 (transfer 0x6fdec033c000)
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x6fdec033c000)
* Connection failed
* connect to 127.0.0.1 port 12500 failed: Connection refused
* Failed to connect to 127.0.0.1 port 12500: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 127.0.0.1 port 12500: Connection refused

I am using
NetBSD NetBSD 8.1 (GENERIC) #0
machine amd64

ghost · 2019-09-16T02:23:16Z

I tried Both fill all for firefox and socks5 only but keep in your mind I am running proxy.py on one machine and firefox on different machine using this command

python3.6 proxy.py --port 12500 --ipv4 --hostname 0.0.0.0

and configure firefox IP:12500

abhinavsingh · 2019-09-16T02:34:01Z

Thanks for the info.

Running proxy.py and Firefox on separate machines should not matter, I expect things to work fine. I see you are also using --ipv4 --hostname 0.0.0.0 options, so proxy.py should be reachable from an external machine on the network.

However, I don't quite understand your curl output. Somehow, you are unable to reach proxy.py via curl.

Are you sure you had proxy.py started before running the curl command?
Were you running curl command on the same machine where proxy.py was running, if not try with machine IP.

If both 1) and 2) are OK, then its obvious there is something else prohibiting connections to proxy.py, may be a firewall?

Also, you should NOT setup Firefox to use proxy.py as SOCKS or FTP proxy.

ghost · 2019-09-16T02:43:50Z

corrected

curl -v -x 0.0.0.0:12500 http://httpbin.org/get

* Expire in 0 ms for 6 (transfer 0x7e349ed3c000)
*   Trying 0.0.0.0...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x7e349ed3c000)
* Connected to 0.0.0.0 (127.0.0.1) port 12500 (#0)
> GET http://httpbin.org/get HTTP/1.1
> Host: httpbin.org
> User-Agent: curl/7.64.0
> Accept: */*
> Proxy-Connection: Keep-Alive

I configured firefox as you mentioned above exactly moreover I tried to fill ssl ftp
no difference in both cases and not working

ghost · 2019-09-16T02:49:31Z

$ curl -v -x 0.0.0.0:12500 http://httpbin.org/get

* Expire in 0 ms for 6 (transfer 0x7c7e6bb3c000)
*   Trying 0.0.0.0...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x7c7e6bb3c000)
* Connected to 0.0.0.0 (127.0.0.1) port 12500 (#0)
> GET http://httpbin.org/get HTTP/1.1
> Host: httpbin.org
> User-Agent: curl/7.64.0
> Accept: */*
> Proxy-Connection: Keep-Alive
>
* Empty reply from server
* Connection #0 to host 0.0.0.0 left intact
curl: (52) Empty reply from server

abhinavsingh · 2019-09-16T03:04:16Z

So now we have curl connecting to proxy.py, however you get an empty response. Frankly I haven't personally tried proxy.py on NetBSD yet, so we might be chasing some bug here. To help debug further can you try this:

Start proxy.py with debug level logging in Terminal 1

python3.6 proxy.py --port 12500 --ipv4 --hostname 0.0.0.0 --log-level d

On Terminal 2, execute the same curl command again

curl -v -x 0.0.0.0:12500 http://httpbin.org/get

Share the exception / stacktrace printed on Terminal 1.

abhinavsingh · 2019-09-16T03:05:25Z

@mbatrawi I said share "exception / stacktrace", infact share everything printed on Terminal 1 to help understand the flow better. Thank you!!!

abhinavsingh · 2019-09-16T03:13:29Z

Searched for RuntimeError: received 0 items of ancdata and found few similar issues in other Python packages. This comment suggests increasing ulimit on the system fixed their issue.

proxy.py automatically tries to set ulimit to 1024. If you start proxy.py with debug logging, you should see something like this:

$ ./proxy.py --log-level d
2019-09-15 20:09:22,894 - DEBUG - pid:45584 - set_open_file_limit:1132 - Open file descriptor soft limit set to 1024

Try increasing the ulimit to 4096 for your system and see if that helps. Pass following command line option too --open-file-limit 4096 when starting proxy.py

ghost · 2019-09-16T04:23:28Z

2019-09-16 04:17:53,993 - INFO - pid:6667 - load_plugins:1151 - Loaded plugin <class 'proxy.HttpProxyPlugin'>
2019-09-16 04:17:53,994 - INFO - pid:6667 - setup:262 - Starting 8 workers
2019-09-16 04:17:54,127 - INFO - pid:6667 - run:225 - Started server on 0.0.0.0:12500
2019-09-16 04:19:08,077 - DEBUG - pid:6667 - handle:276 - Dispatched client request to worker id 0
Process Worker-1:
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "proxy.py", line 309, in run
    op, payload = self.work_queue.get(True, 1)
  File "/usr/pkg/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 239, in _rebuild_socket
    fd = df.detach()
  File "/usr/pkg/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
2019-09-16 04:19:46,803 - DEBUG - pid:6667 - handle:276 - Dispatched client request to worker id 1
Process Worker-2:
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "proxy.py", line 309, in run
    op, payload = self.work_queue.get(True, 1)
  File "/usr/pkg/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 239, in _rebuild_socket
    fd = df.detach()
  File "/usr/pkg/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
2019-09-16 04:19:49,786 - DEBUG - pid:6667 - handle:276 - Dispatched client request to worker id 2
Process Worker-3:
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "proxy.py", line 309, in run
    op, payload = self.work_queue.get(True, 1)
  File "/usr/pkg/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 239, in _rebuild_socket
    fd = df.detach()
  File "/usr/pkg/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
2019-09-16 04:19:52,762 - DEBUG - pid:6667 - handle:276 - Dispatched client request to worker id 3
Process Worker-4:
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "proxy.py", line 309, in run
    op, payload = self.work_queue.get(True, 1)
  File "/usr/pkg/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 239, in _rebuild_socket
    fd = df.detach()
  File "/usr/pkg/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
2019-09-16 04:19:56,616 - DEBUG - pid:6667 - handle:276 - Dispatched client request to worker id 4
2019-09-16 04:19:56,616 - DEBUG - pid:6667 - handle:276 - Dispatched client request to worker id 5
Process Worker-5:
Process Worker-6:
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "proxy.py", line 309, in run
    op, payload = self.work_queue.get(True, 1)
  File "/usr/pkg/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 239, in _rebuild_socket
    fd = df.detach()
  File "/usr/pkg/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/pkg/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "proxy.py", line 309, in run
    op, payload = self.work_queue.get(True, 1)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/pkg/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 239, in _rebuild_socket
    fd = df.detach()
  File "/usr/pkg/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
RuntimeError: received 0 items of ancdata
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
2019-09-16 04:19:57,522 - DEBUG - pid:6667 - handle:276 - Dispatched client request to worker id 6
Process Worker-7:
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "proxy.py", line 309, in run
    op, payload = self.work_queue.get(True, 1)
  File "/usr/pkg/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 239, in _rebuild_socket
    fd = df.detach()
  File "/usr/pkg/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
2019-09-16 04:19:59,000 - DEBUG - pid:6667 - handle:276 - Dispatched client request to worker id 7
Process Worker-8:
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "proxy.py", line 309, in run
    op, payload = self.work_queue.get(True, 1)
  File "/usr/pkg/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 239, in _rebuild_socket
    fd = df.detach()
  File "/usr/pkg/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
2019-09-16 04:19:59,031 - DEBUG - pid:6667 - handle:276 - Dispatched client request to worker id 0
2019-09-16 04:19:59,854 - DEBUG - pid:6667 - handle:276 - Dispatched client request to worker id 1
2019-09-16 04:19:59,855 - DEBUG - pid:6667 - handle:276 - Dispatched client request to worker id 2
2019-09-16 04:20:07,497 - DEBUG - pid:6667 - handle:276 - Dispatched client request to worker id 3
2019-09-16 04:20:39,223 - DEBUG - pid:6667 - handle:276 - Dispatched client request to worker id 4

ghost · 2019-09-16T04:26:06Z

curl -v -x 0.0.0.0:12500 http://httpbin.org/get

* Expire in 0 ms for 6 (transfer 0x7baad213c000)
*   Trying 0.0.0.0...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x7baad213c000)
* Connected to 0.0.0.0 (127.0.0.1) port 12500 (#0)
> GET http://httpbin.org/get HTTP/1.1
> Host: httpbin.org
> User-Agent: curl/7.64.0
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
* Empty reply from server
* Connection #0 to host 0.0.0.0 left intact
curl: (52) Empty reply from server

ghost · 2019-09-16T10:57:36Z

--hostname 0.0.0.0 --log-level d --open-file-limit 4096  <
2019-09-16 10:50:20,001 - INFO - pid:5539 - load_plugins:1151 - Loaded plugin <class 'proxy.HttpProxyPlugin'>
2019-09-16 10:50:20,001 - INFO - pid:5539 - setup:262 - Starting 8 workers
2019-09-16 10:50:20,134 - INFO - pid:5539 - run:225 - Started server on 0.0.0.0:12500
2019-09-16 10:50:53,993 - DEBUG - pid:5539 - handle:276 - Dispatched client request to worker id 0
Process Worker-1:
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "proxy.py", line 309, in run
    op, payload = self.work_queue.get(True, 1)
  File "/usr/pkg/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 239, in _rebuild_socket
    fd = df.detach()
  File "/usr/pkg/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
2019-09-16 10:50:57,064 - DEBUG - pid:5539 - handle:276 - Dispatched client request to worker id 1
Process Worker-2:
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "proxy.py", line 309, in run
    op, payload = self.work_queue.get(True, 1)
  File "/usr/pkg/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 239, in _rebuild_socket
    fd = df.detach()
  File "/usr/pkg/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
2019-09-16 10:50:59,795 - DEBUG - pid:5539 - handle:276 - Dispatched client request to worker id 2
Process Worker-3:
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "proxy.py", line 309, in run
    op, payload = self.work_queue.get(True, 1)
  File "/usr/pkg/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 239, in _rebuild_socket
    fd = df.detach()
  File "/usr/pkg/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
2019-09-16 10:51:02,100 - DEBUG - pid:5539 - handle:276 - Dispatched client request to worker id 3
Process Worker-4:
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "proxy.py", line 309, in run
    op, payload = self.work_queue.get(True, 1)
  File "/usr/pkg/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 239, in _rebuild_socket
    fd = df.detach()
  File "/usr/pkg/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
2019-09-16 10:51:04,542 - DEBUG - pid:5539 - handle:276 - Dispatched client request to worker id 4
Process Worker-5:
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "proxy.py", line 309, in run
    op, payload = self.work_queue.get(True, 1)
  File "/usr/pkg/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 239, in _rebuild_socket
    fd = df.detach()
  File "/usr/pkg/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
2019-09-16 10:51:07,111 - DEBUG - pid:5539 - handle:276 - Dispatched client request to worker id 5
Process Worker-6:
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "proxy.py", line 309, in run
    op, payload = self.work_queue.get(True, 1)
  File "/usr/pkg/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 239, in _rebuild_socket
    fd = df.detach()
  File "/usr/pkg/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
2019-09-16 10:51:19,291 - DEBUG - pid:5539 - handle:276 - Dispatched client request to worker id 6
Process Worker-7:
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "proxy.py", line 309, in run
    op, payload = self.work_queue.get(True, 1)
  File "/usr/pkg/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 239, in _rebuild_socket
    fd = df.detach()
  File "/usr/pkg/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
2019-09-16 10:51:20,177 - DEBUG - pid:5539 - handle:276 - Dispatched client request to worker id 7
Process Worker-8:
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "proxy.py", line 309, in run
    op, payload = self.work_queue.get(True, 1)
  File "/usr/pkg/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 239, in _rebuild_socket
    fd = df.detach()
  File "/usr/pkg/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
2019-09-16 10:51:49,910 - DEBUG - pid:5539 - handle:276 - Dispatched client request to worker id 0
2019-09-16 10:51:50,724 - DEBUG - pid:5539 - handle:276 - Dispatched client request to worker id 1

abhinavsingh · 2019-09-16T17:25:47Z

Thanks @mbatrawi for the logs, it really helps. I believe we have a bug related to how client sockets are passed around between processes, I see various projects hitting similar exception when trying to pass sockets between processes. Amazingly this bug only surfaces on NetBSD, I have personally tried develop branch on MacOS, Ubuntu and Windows without any errors.

proxy.py develop branch introduced the concept of multiple worker processes. proxy.py stable version which didn't use multiprocessing should work fine for you, simply install using pip install proxy.py==0.3.

I'll update the thread once I have a hot-fix in.

ghost · 2019-09-16T18:22:53Z

can you send me stable file as zip file

abhinavsingh · 2019-09-16T18:30:22Z

Master branch contains the stable version, download it here https://github.com/abhinavsingh/proxy.py/archive/master.zip

ghost · 2019-09-16T20:01:20Z

stable one is perfect
thanks

Possible solution for #68

abhinavsingh · 2019-09-16T21:40:30Z

@mbatrawi Thanks for raising this. We might have a fix for the develop branch, could you give following a try https://github.com/abhinavsingh/proxy.py/archive/pipes.zip Thanks :)

* Use pipes instead of queues. Possible solution for #68 * Handle ConnectionRefusedError * Close corresponding pipes on shutdown * Abstract classes for plugins. * Add github workflows * Use pytest for github workflow * Add Windows/Mac workflows * Use os.matrix * 3.8 and 3.9 probably dont exists yet * Do not fail-fast, also python 3.5 seems to be throwing syntax error for typing * autopep8 * Disable windows-latest workflow which seems to hang

ghost · 2019-09-17T09:14:03Z

2019-09-17 09:00:23,788 - INFO - pid:23291 - load_plugins:1272 - Loaded plugin <class 'proxy.HttpProxyPlugin'>
2019-09-17 09:00:23,789 - INFO - pid:23291 - setup:282 - Starting 8 workers
2019-09-17 09:00:23,819 - INFO - pid:23291 - run:239 - Started server on 0.0.0.0:12500
Process Worker-1:
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "proxy.py", line 332, in run
    op, payload = self.work_queue.recv()
  File "/usr/pkg/lib/python3.6/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 239, in _rebuild_socket
    fd = df.detach()
  File "/usr/pkg/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/pkg/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata

ghost · 2019-09-17T09:36:33Z

Not working

abhinavsingh · 2019-09-17T16:03:59Z

@mbatrawi Thanks for verifying this quickly. Probably should get a NetBSD image to give it a try myself :) Unfortunately, installation via VirtualBox didn't really worked out for me.

ghost · 2019-09-17T17:01:31Z

do you have good knowledge about c language ?

ghost · 2019-09-17T17:05:52Z

check this one and if you can update it or convert to python
https://github.com/unquietwiki/stone

ghost · 2019-09-18T18:04:50Z

python3.6 proxy.py --port 12500 --hostname 0.0.0.0  --log-level d
2019-09-18 21:59:36,079 - INFO - pid:23192 - load_plugins:1151 - Loaded plugin <class 'proxy.HttpProxyPlugin'>
2019-09-18 21:59:36,080 - INFO - pid:23192 - setup:262 - Starting 4 workers
2019-09-18 21:59:36,111 - ERROR - pid:23192 - run:233 - Exception while running the server gaierror(-9, 'Address family for hostname not supported')
Traceback (most recent call last):
  File "proxy.py", line 223, in run
    self.socket.bind((self.hostname, self.port))
socket.gaierror: [Errno -9] Address family for hostname not supported
2019-09-18 21:59:36,114 - INFO - pid:23192 - shutdown:282 - Shutting down 4 workers
2019-09-18 21:59:36,118 - INFO - pid:23192 - run:236 - Closing server socket

system

uname -a
Linux triton 4.14.65-gentoo-bs #1 SMP Sun Oct 28 14:35:37 CET 2018 x86_64 Intel(R) Xeon(R) CPU X3440 @ 2.53GHz GenuineIntel GNU/Linux

last update proxy.py

abhinavsingh · 2019-09-19T03:02:03Z

@mbatrawi proxy.py development branch defaults to IPv6 addressing. If you are overriding --hostname 0.0.0.0 you must also pass --ipv4 flag.

I'll create a feature request to auto-detect this internally to avoid passing --ipv4 flag :)

abhinavsingh · 2019-09-22T08:11:13Z

Closing this since looks like issue is more with your browser configuration. Per #71 (comment) you are able to browser web via curl over the network. I believe once you have the browser config correct, it should work out just fine. Thank you!!!

abhinavsingh added Awaiting Response Waiting for more information / response from issue creator Question Questions related to proxy server labels Sep 16, 2019

abhinavsingh changed the title ~~httproxy~~ Proxy.py with Firefox Sep 16, 2019

abhinavsingh changed the title ~~Proxy.py with Firefox~~ Proxy.py with Firefox on NetBSD Sep 16, 2019

abhinavsingh added Bug Bug report in proxy server In Develop Issues affecting only the development branch and removed Awaiting Response Waiting for more information / response from issue creator Question Questions related to proxy server labels Sep 16, 2019

abhinavsingh added a commit that referenced this issue Sep 16, 2019

Use pipes instead of queues.

ef676b0

Possible solution for #68

abhinavsingh self-assigned this Sep 17, 2019

abhinavsingh mentioned this issue Sep 19, 2019

Auto-detect IPv4 / IPv6 listen based upon --hostname arg #71

Closed

abhinavsingh closed this as completed Sep 22, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Proxy.py with Firefox on NetBSD #68

Proxy.py with Firefox on NetBSD #68

ghost commented Sep 15, 2019 •

edited by abhinavsingh

Loading

abhinavsingh commented Sep 16, 2019

abhinavsingh commented Sep 16, 2019

ghost commented Sep 16, 2019 •

edited by abhinavsingh

Loading

ghost commented Sep 16, 2019 •

edited by abhinavsingh

Loading

abhinavsingh commented Sep 16, 2019

ghost commented Sep 16, 2019 •

edited by abhinavsingh

Loading

ghost commented Sep 16, 2019 •

edited by abhinavsingh

Loading

abhinavsingh commented Sep 16, 2019

abhinavsingh commented Sep 16, 2019

abhinavsingh commented Sep 16, 2019 •

edited

Loading

ghost commented Sep 16, 2019 •

edited by abhinavsingh

Loading

ghost commented Sep 16, 2019 •

edited by abhinavsingh

Loading

ghost commented Sep 16, 2019 •

edited by abhinavsingh

Loading

abhinavsingh commented Sep 16, 2019

ghost commented Sep 16, 2019

abhinavsingh commented Sep 16, 2019

ghost commented Sep 16, 2019

abhinavsingh commented Sep 16, 2019

ghost commented Sep 17, 2019 •

edited by abhinavsingh

Loading

ghost commented Sep 17, 2019

abhinavsingh commented Sep 17, 2019

ghost commented Sep 17, 2019

ghost commented Sep 17, 2019

ghost commented Sep 18, 2019 •

edited by abhinavsingh

Loading

abhinavsingh commented Sep 19, 2019

abhinavsingh commented Sep 22, 2019

Proxy.py with Firefox on NetBSD #68

Proxy.py with Firefox on NetBSD #68

Comments

ghost commented Sep 15, 2019 • edited by abhinavsingh Loading

abhinavsingh commented Sep 16, 2019

abhinavsingh commented Sep 16, 2019

ghost commented Sep 16, 2019 • edited by abhinavsingh Loading

ghost commented Sep 16, 2019 • edited by abhinavsingh Loading

abhinavsingh commented Sep 16, 2019

ghost commented Sep 16, 2019 • edited by abhinavsingh Loading

ghost commented Sep 16, 2019 • edited by abhinavsingh Loading

abhinavsingh commented Sep 16, 2019

abhinavsingh commented Sep 16, 2019

abhinavsingh commented Sep 16, 2019 • edited Loading

ghost commented Sep 16, 2019 • edited by abhinavsingh Loading

ghost commented Sep 16, 2019 • edited by abhinavsingh Loading

ghost commented Sep 16, 2019 • edited by abhinavsingh Loading

abhinavsingh commented Sep 16, 2019

ghost commented Sep 16, 2019

abhinavsingh commented Sep 16, 2019

ghost commented Sep 16, 2019

abhinavsingh commented Sep 16, 2019

ghost commented Sep 17, 2019 • edited by abhinavsingh Loading

ghost commented Sep 17, 2019

abhinavsingh commented Sep 17, 2019

ghost commented Sep 17, 2019

ghost commented Sep 17, 2019

ghost commented Sep 18, 2019 • edited by abhinavsingh Loading

abhinavsingh commented Sep 19, 2019

abhinavsingh commented Sep 22, 2019

ghost commented Sep 15, 2019 •

edited by abhinavsingh

Loading

ghost commented Sep 16, 2019 •

edited by abhinavsingh

Loading

ghost commented Sep 16, 2019 •

edited by abhinavsingh

Loading

ghost commented Sep 16, 2019 •

edited by abhinavsingh

Loading

ghost commented Sep 16, 2019 •

edited by abhinavsingh

Loading

abhinavsingh commented Sep 16, 2019 •

edited

Loading

ghost commented Sep 16, 2019 •

edited by abhinavsingh

Loading

ghost commented Sep 16, 2019 •

edited by abhinavsingh

Loading

ghost commented Sep 16, 2019 •

edited by abhinavsingh

Loading

ghost commented Sep 17, 2019 •

edited by abhinavsingh

Loading

ghost commented Sep 18, 2019 •

edited by abhinavsingh

Loading