bpo-28724: Add methods send_fds, recv_fds to the socket module #12889

nanjekyejoannah · 2019-04-20T22:44:32Z

I have added send_fds(), recv_fds() to the socket module.

https://bugs.python.org/issue28724

Doc/library/socket.rst

Lib/socket.py

Lib/test/test_socket.py

Lib/socket.py

bedevere-bot · 2019-05-04T22:50:31Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

pablogsal · 2019-05-10T19:34:35Z

Lib/test/test_socket.py

+    @requireAttrs(socket.socket, 'send_fds')
+    def testSendAndRecvFds(self):
+        fds = []
+        for i in range(1):


Why we need a loop of 1 iteration?

I want to return a list of 1 file descriptors for created files.

But you don't need a loop for that, right? Just substitute i by 0. I am missing something?

Lib/test/test_socket.py

Misc/NEWS.d/next/Library/2019-05-10-19-27-07.bpo-28724.rHcm24.rst

nanjekyejoannah · 2019-05-27T15:58:38Z

Needs, Mac OS fixes. Will get a Mac to test this later.

nanjekyejoannah · 2019-08-24T16:41:28Z

@pablogsal , Please help me confirm If the tests pass on MacOS now. I cant access a Mac now.

nanjekyejoannah · 2019-08-24T16:42:09Z

I have made the requested changes; please review again

bedevere-bot · 2019-08-24T16:42:13Z

Thanks for making the requested changes!

@tiran, @pablogsal, @vstinner, @ncoghlan: please review the changes made to this pull request.

vstinner · 2019-08-26T22:32:11Z

Lib/socket.py

+    import array
+
+    def send_fds(sock, buffers, fds, flags=0, address=None):
+        """ send_fds(sock, buffers, fds[, flags[, address]]) -> socket object


It seems like sendmsg() returns an integer, not a socket object.

vstinner · 2019-08-26T22:33:08Z

Lib/socket.py

+    import array
+
+    def recv_fds(sock, bufsize, maxfds, flags=0):
+        """ recv_fds(sock, bufsize, maxfds[, flags]) -> (socket object, socket object)


The doc of the return value looks wrong. The code uses a tuple of 4 times: return msg, list(fds), flags, addr.

vstinner · 2019-08-26T22:33:25Z

Lib/socket.py

+    def recv_fds(sock, bufsize, maxfds, flags=0):
+        """ recv_fds(sock, bufsize, maxfds[, flags]) -> (socket object, socket object)
+
+        receive up to maxfds file descriptors returning the message


Upper case for the first word of the doc?

Suggested change

receive up to maxfds file descriptors returning the message

Receive up to maxfds file descriptors returning the message

vstinner · 2019-08-26T22:34:51Z

Lib/test/test_socket.py

+        with sock1, sock2:
+            socket.send_fds(sock1, [MSG], fds)
+            # request more data and file descriptors than expected
+            msg, fds2, flags, addr  = socket.recv_fds(sock2, len(MSG) * 2, len(fds) * 2)


nitpick (PEP 8): there is an useless space

Suggested change

msg, fds2, flags, addr = socket.recv_fds(sock2, len(MSG) * 2, len(fds) * 2)

msg, fds2, flags, addr = socket.recv_fds(sock2, len(MSG) * 2, len(fds) * 2)

vstinner · 2019-08-26T22:37:58Z

Doc/library/socket.rst

+   Receive up to *maxfds* file descriptors. Return ``(msg, list(fds), flags, addr)``. Consult
+   :meth:`recvmsg` for the documentation of these parameters.
+
+   .. availability:: Unix supporting :meth:`~socket.sendmsg` and :const:`SCM_RIGHTS` mechanism.


it requires recvmsg() instead of sendmsg(), no?

vstinner · 2019-08-26T22:38:32Z

Doc/library/socket.rst

+.. method:: socket.send_fds(sock, buffers, fds[, flags[, address]])
+
+   Send the list of file descriptors *fds* over an :const:`AF_UNIX` socket.
+   Consult :meth:`sendmsg` for the documentation of these parameters.


You should document at least what is the fds parameter. I suggest to document it as sequence of file descriptors. Is it possible to send 0 file descriptor?

What is the return value?

Does someone know if it's possible that the call send only a few file descriptors, but not all of them? What if it's called with 100 000 file descriptors for example?

What is the return value?

The return type for sendmsg() is int AFAIK. Am not sure about these other edge cases but I can look.

I tested to send 1000 file descriptors: sendmsg() fails with "OSError: [Errno 22] Invalid argument".

For partial send, it can happen, as in socket.send(). In that case, according to https://stackoverflow.com/questions/38431713/partial-read-write-issue-in-sendmsg-recvmsg the file descriptors are already sent and should be be send again, socket.sendmsg() should be called again, but only with buffers (adjusted to the offset equal to the send_fds() result).

In short, you don't have to do anything. I'm not comfortable to document how to handle partial send since I'm not sure that the behavior is portable.

sendmsg() doesn't warn about partial send. Maybe we should copy send() warning into sendmsg() and send_fds(): "Applications are responsible for checking that all data has been sent; if only some of the data was transmitted, the application needs to attempt delivery of the remaining data."

bedevere-bot · 2019-08-26T22:41:09Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

nanjekyejoannah · 2019-08-28T14:39:47Z

I have made the requested changes; please review again

bedevere-bot · 2019-08-28T14:39:51Z

Thanks for making the requested changes!

@ncoghlan, @pablogsal, @vstinner, @tiran: please review the changes made to this pull request.

vstinner

The test is skipped. When I enabled manually the test, it failed on Linux :-(

vstinner · 2019-08-28T21:20:29Z

Doc/library/socket.rst

+.. method:: socket.send_fds(sock, buffers, fds[, flags[, address]])
+
+   Send the list of file descriptors *fds* over an :const:`AF_UNIX` socket.
+   Consult :meth:`sendmsg` for the documentation of these parameters.


I tested to send 1000 file descriptors: sendmsg() fails with "OSError: [Errno 22] Invalid argument".

For partial send, it can happen, as in socket.send(). In that case, according to https://stackoverflow.com/questions/38431713/partial-read-write-issue-in-sendmsg-recvmsg the file descriptors are already sent and should be be send again, socket.sendmsg() should be called again, but only with buffers (adjusted to the offset equal to the send_fds() result).

In short, you don't have to do anything. I'm not comfortable to document how to handle partial send since I'm not sure that the behavior is portable.

sendmsg() doesn't warn about partial send. Maybe we should copy send() warning into sendmsg() and send_fds(): "Applications are responsible for checking that all data has been sent; if only some of the data was transmitted, the application needs to attempt delivery of the remaining data."

vstinner · 2019-08-28T21:22:47Z

Lib/test/test_socket.py

+@requireAttrs(socket, "send_fds")
+@requireAttrs(socket, "recv_fds")
+@requireAttrs(socket, "AF_UNIX")
+class SendRecvFdsTests(unittest.TestCase):


Oh, this test case must be added manually into test_main(). Currently, the test is not run...

I added it manually and... the test fails :-( The test writes into the read end of the pipe, and reads from the write end of the pipe. It should be the opposite. Maybe on FreeBSD, pipe ends are not directional, but it seems to be the case on Linux.

So the test should be fixed on Linux a well.

Please move this class before test_main().

vstinner · 2019-08-28T21:24:12Z

Lib/socket.py

+            _socket.CMSG_LEN(maxfds * fds.itemsize))
+        for cmsg_level, cmsg_type, cmsg_data in ancdata:
+            if (cmsg_level == _socket.SOL_SOCKET and cmsg_type == _socket.SCM_RIGHTS):
+                # Append data, ignoring any truncated integers at the end.


@nanjekyejoannah: Would you mind to document this behavior in recv_fds() documention?

vstinner · 2019-08-28T21:26:50Z

Lib/socket.py

@@ -12,6 +12,8 @@
 socket() -- create a new socket object
 socketpair() -- create a pair of new socket objects [*]
 fromfd() -- create a socket object from an open file descriptor [*]
+send_fds() -- Send file descriptor to the socket.
+recv_fds() -- Recieve file descriptor from the socket.


I suggest to replace "file descriptor" with "file descriptors" for both functions here (add an S), as in their name, since they can send and receive more than one file descriptor ;-)

bedevere-bot · 2019-08-28T21:27:30Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

vstinner · 2019-09-09T10:29:57Z

The test is skipped. When I enabled manually the test, it failed on Linux :-(

test_main() is awful... and it's no longer needed: unittest.main() can be used instead.

I created https://bugs.python.org/issue38063 "Modify test_socket.py to use unittest test discovery" and I wrote a PR to implement it.

nanjekyejoannah · 2019-09-11T15:55:14Z

@vstinner

Test passes with your fix.

Ran 580 tests in 24.587s

OK (skipped=118)

== Tests result: SUCCESS ==

1 test OK.

Total duration: 24 sec 956 ms
Tests result: SUCCES

Also the test ir running:
testSendAndRecvFds (test.test_socket.SendRecvFdsTests) ... ok

vstinner · 2019-09-11T17:09:29Z

I tested manually commit 1ac7fbf on Linux and FreeBSD. I merged the PR with the master branch, make && ./python -m test test_socket -m SendRecvFdsTests -v . The test pass on Linux and FreeBSD. Good!

According to Azure Pipelines: test_socket also pass on macOS (and the test is no longer skipped on macOS).

test_socket pass on Windows as well, even if the test is skipped on Windows.

vstinner · 2019-09-11T17:17:43Z

Thanks Shinya Okano for the original patch.

Well done Joannah @nanjekyejoannah! Thanks for your tenacity :-) This PR has 100 comments and 27 commits which shows the complexity of the feature.

Honestly, I'm not 100% happy with current documentation, but I chose to merge the PR anyway. Please leave https://bugs.python.org/issue28724 open until the documentation is completed to mention corner cases like partial send (similar to sock.send vs sock.sendall).

…thonGH-12889) The socket module now has the socket.send_fds() and socket.recv.fds() functions. Contributed by Joannah Nanjekye, Shinya Okano (original patch) and Victor Stinner. Co-Authored-By: Victor Stinner <[email protected]>

the-knights-who-say-ni added the CLA signed label Apr 20, 2019

bedevere-bot added the awaiting review label Apr 20, 2019

nanjekyejoannah changed the title ~~Add method send_fds, recv_fds to the socket module~~ bpo-28724: Add method send_fds, recv_fds to the socket module Apr 20, 2019

nanjekyejoannah changed the title ~~bpo-28724: Add method send_fds, recv_fds to the socket module~~ bpo-28724: Add methodS send_fds, recv_fds to the socket module Apr 20, 2019

nanjekyejoannah changed the title ~~bpo-28724: Add methodS send_fds, recv_fds to the socket module~~ bpo-28724: Add methods send_fds, recv_fds to the socket module Apr 22, 2019

ncoghlan requested changes May 4, 2019

View reviewed changes

bedevere-bot added awaiting changes and removed awaiting review labels May 4, 2019

nanjekyejoannah requested review from 1st1, asvetlov, ericvsmith, gpshead, pablogsal, rhettinger and a team as code owners May 6, 2019 17:09

nanjekyejoannah force-pushed the issue28724 branch 3 times, most recently from 40d7ea4 to f3e0822 Compare May 10, 2019 19:26

pablogsal reviewed May 10, 2019

View reviewed changes

nanjekyejoannah requested review from brettcannon, encukou, ericsnowcurrently, gvanrossum, ilevkivskyi, terryjreedy, tiran and warsaw as code owners May 30, 2019 15:13

Do not skip Macos in tests

0f03c42

bedevere-bot added awaiting change review and removed awaiting changes labels Aug 24, 2019

vstinner requested changes Aug 26, 2019

View reviewed changes

bedevere-bot removed the awaiting change review label Aug 26, 2019

bedevere-bot added the awaiting changes label Aug 26, 2019

Fix doc strings

bf67bd8

bedevere-bot added awaiting change review and removed awaiting changes labels Aug 28, 2019

vstinner requested changes Aug 28, 2019

View reviewed changes

bedevere-bot added awaiting changes and removed awaiting change review labels Aug 28, 2019

Fix test

1ac7fbf

vstinner merged commit 8d120f7 into python:master Sep 11, 2019

bedevere-bot removed the awaiting changes label Sep 11, 2019

nanjekyejoannah deleted the issue28724 branch September 11, 2019 17:13

ezio-melotti mentioned this pull request May 31, 2022

Remove execution bit from socket-related files. #93367

Closed

	receive up to maxfds file descriptors returning the message
	Receive up to maxfds file descriptors returning the message

	msg, fds2, flags, addr = socket.recv_fds(sock2, len(MSG) * 2, len(fds) * 2)
	msg, fds2, flags, addr = socket.recv_fds(sock2, len(MSG) * 2, len(fds) * 2)

Uh oh!

bpo-28724: Add methods send_fds, recv_fds to the socket module #12889

bpo-28724: Add methods send_fds, recv_fds to the socket module #12889

Uh oh!

Conversation

nanjekyejoannah commented Apr 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bedevere-bot commented May 4, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

nanjekyejoannah commented May 27, 2019

Uh oh!

nanjekyejoannah commented Aug 24, 2019

Uh oh!

nanjekyejoannah commented Aug 24, 2019

Uh oh!

bedevere-bot commented Aug 24, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Aug 26, 2019

Uh oh!

nanjekyejoannah commented Aug 28, 2019

Uh oh!

bedevere-bot commented Aug 28, 2019

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Aug 28, 2019

Uh oh!

vstinner commented Sep 9, 2019

Uh oh!

nanjekyejoannah commented Sep 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner commented Sep 11, 2019

Uh oh!

vstinner commented Sep 11, 2019

Uh oh!

Uh oh!

nanjekyejoannah commented Apr 20, 2019 •

edited

Loading

nanjekyejoannah commented Sep 11, 2019 •

edited

Loading