Skip to content

bpo-28724: Add methods send_fds, recv_fds to the socket module #12889

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 27 commits into from
Sep 11, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
fd7b901
Add methods send_fds, recv_fds to the socket module
nanjekyejoannah May 30, 2019
2558c5d
📜🤖 Added by blurb_it.
blurb-it[bot] May 30, 2019
dfdeb76
Fix tests on MacOS
nanjekyejoannah May 30, 2019
bc58424
Merge branch 'issue28724' of https://github.com/nanjekyejoannah/cpyth…
nanjekyejoannah May 30, 2019
cc6973c
Attempt to fix MacOS
nanjekyejoannah May 30, 2019
5050e59
MacOS fix
nanjekyejoannah May 30, 2019
fe8ff1a
Mac OS fix
nanjekyejoannah May 30, 2019
e98918f
fix Mac OS
nanjekyejoannah May 30, 2019
c3fcd6f
Attempt again
nanjekyejoannah May 30, 2019
d497c15
keep the function signatures close to the original ones
nanjekyejoannah May 31, 2019
035845a
make return types consistent
nanjekyejoannah May 31, 2019
e6d22e0
Fix freebsd
nanjekyejoannah Jun 1, 2019
f84d6db
Remove unused global sentinel .
nanjekyejoannah Jun 3, 2019
07b05dc
Update documentation
nanjekyejoannah Jun 6, 2019
8b96493
Fix docs
nanjekyejoannah Jun 6, 2019
7c46e7a
Update Doc/library/socket.rst
nanjekyejoannah Jun 6, 2019
2c32da1
Add Victor's test and update docs
Jun 13, 2019
75bffbc
no need for new file
nanjekyejoannah Jun 13, 2019
ba29ca2
Update 3.8.rst
nanjekyejoannah Jun 13, 2019
83e422d
update 3.9.rst
Jun 13, 2019
910461b
Delete 3.9.rst
nanjekyejoannah Jun 13, 2019
b134b48
Update test_socket.py
nanjekyejoannah Jun 13, 2019
ee950f8
Update test_socket.py
nanjekyejoannah Jun 13, 2019
36e157d
fix space
Jun 13, 2019
0f03c42
Do not skip Macos in tests
nanjekyejoannah Aug 24, 2019
bf67bd8
Fix doc strings
nanjekyejoannah Aug 28, 2019
1ac7fbf
Fix test
nanjekyejoannah Sep 11, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions Doc/library/socket.rst
100644 → 100755
Original file line number Diff line number Diff line change
Expand Up @@ -1564,6 +1564,29 @@ to sockets.

.. versionadded:: 3.6

.. method:: socket.send_fds(sock, buffers, fds[, flags[, address]])

Send the list of file descriptors *fds* over an :const:`AF_UNIX` socket.
The *fds* parameter is a sequence of file descriptors.
Consult :meth:`sendmsg` for the documentation of these parameters.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should document at least what is the fds parameter. I suggest to document it as sequence of file descriptors. Is it possible to send 0 file descriptor?

What is the return value?

Does someone know if it's possible that the call send only a few file descriptors, but not all of them? What if it's called with 100 000 file descriptors for example?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the return value?

The return type for sendmsg() is int AFAIK. Am not sure about these other edge cases but I can look.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested to send 1000 file descriptors: sendmsg() fails with "OSError: [Errno 22] Invalid argument".

For partial send, it can happen, as in socket.send(). In that case, according to https://stackoverflow.com/questions/38431713/partial-read-write-issue-in-sendmsg-recvmsg the file descriptors are already sent and should be be send again, socket.sendmsg() should be called again, but only with buffers (adjusted to the offset equal to the send_fds() result).

In short, you don't have to do anything. I'm not comfortable to document how to handle partial send since I'm not sure that the behavior is portable.

sendmsg() doesn't warn about partial send. Maybe we should copy send() warning into sendmsg() and send_fds(): "Applications are responsible for checking that all data has been sent; if only some of the data was transmitted, the application needs to attempt delivery of the remaining data."


.. availability:: Unix supporting :meth:`~socket.sendmsg` and :const:`SCM_RIGHTS` mechanism.

.. versionadded:: 3.9

.. method:: socket.recv_fds(sock, bufsize, maxfds[, flags])

Receive up to *maxfds* file descriptors. Return ``(msg, list(fds), flags, addr)``. Consult
:meth:`recvmsg` for the documentation of these parameters.

.. availability:: Unix supporting :meth:`~socket.recvmsg` and :const:`SCM_RIGHTS` mechanism.

.. versionadded:: 3.9

.. note::

Any truncated integers at the end of the list of file descriptors.

.. method:: socket.sendfile(file, offset=0, count=None)

Send a file until EOF is reached by using high-performance
Expand Down
Empty file modified Doc/whatsnew/3.8.rst
100644 → 100755
Empty file.
36 changes: 36 additions & 0 deletions Lib/socket.py
100644 → 100755
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@
socket() -- create a new socket object
socketpair() -- create a pair of new socket objects [*]
fromfd() -- create a socket object from an open file descriptor [*]
send_fds() -- Send file descriptor to the socket.
recv_fds() -- Recieve file descriptors from the socket.
fromshare() -- create a socket object from data received from socket.share() [*]
gethostname() -- return the current hostname
gethostbyname() -- map a hostname to its IP number
Expand Down Expand Up @@ -463,6 +465,40 @@ def fromfd(fd, family, type, proto=0):
nfd = dup(fd)
return socket(family, type, proto, nfd)

if hasattr(_socket.socket, "sendmsg"):
import array

def send_fds(sock, buffers, fds, flags=0, address=None):
""" send_fds(sock, buffers, fds[, flags[, address]]) -> integer

Send the list of file descriptors fds over an AF_UNIX socket.
"""
return sock.sendmsg(buffers, [(_socket.SOL_SOCKET,
_socket.SCM_RIGHTS, array.array("i", fds))])
__all__.append("send_fds")

if hasattr(_socket.socket, "recvmsg"):
import array

def recv_fds(sock, bufsize, maxfds, flags=0):
""" recv_fds(sock, bufsize, maxfds[, flags]) -> (data, list of file
descriptors, msg_flags, address)

Receive up to maxfds file descriptors returning the message
data and a list containing the descriptors.
"""
# Array of ints
fds = array.array("i")
msg, ancdata, flags, addr = sock.recvmsg(bufsize,
_socket.CMSG_LEN(maxfds * fds.itemsize))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will likely fail on FreeBSD as RFC 3542 requires portable applications to use CMSG_SPACE if I am not mistaken.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should check that this works on FreeBSD to avoid broken buildbots with obscure errors.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pablogsal You mean at test level ?

Copy link
Member

@pablogsal pablogsal Jun 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean that FreeBSD systems running that code can fail and that will likely appear when running the tests that go over this code path.

This is an example of similar scenarios of CMSG_LEN vs CMSG_SPACE:

#9613

I could be wrong with my argument, that's why I think we should check directly on a FreeBSD system.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont have a freeBSD system to confirm this atm. When I do find, I will confirm but someone else can help confirm this. Do you have a FreeBSD to help check this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

@pablogsal pablogsal Jun 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that my suspicions were correct:

test_errors (test.test_socket.SendfileUsingSendfileTest) ... ok
test test_socket failed
test_new_tcp_flags (test.test_socket.TestMSWindowsTCPFlags) ... skipped 'requires Windows'
======================================================================
ERROR: testSendAndRecvFds (test.test_socket.BasicTCPTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/custom.koobs-freebsd10.nondebug/build/Lib/test/test_socket.py", line 2294, in testSendAndRecvFds
    socket.send_fds(sock, [MSG], fds)
  File "/usr/home/buildbot/python/custom.koobs-freebsd10.nondebug/build/Lib/socket.py", line 476, in send_fds
    return sock.sendmsg(buffers, [(_socket.SOL_SOCKET,
OSError: [Errno 22] Invalid argument
======================================================================
ERROR: testSendAndRecvFds (test.test_socket.BasicTCPTest2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/custom.koobs-freebsd10.nondebug/build/Lib/test/test_socket.py", line 2294, in testSendAndRecvFds
    socket.send_fds(sock, [MSG], fds)
  File "/usr/home/buildbot/python/custom.koobs-freebsd10.nondebug/build/Lib/socket.py", line 476, in send_fds
    return sock.sendmsg(buffers, [(_socket.SOL_SOCKET,
OSError: [Errno 22] Invalid argument
----------------------------------------------------------------------
Ran 581 tests in 29.349s

but it could be something also on top of CMSG_LEN

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@koobs Could you please take a look? The new feature is failing on FreeBSD.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can confirm that is the same error as in OSX so it may be easier to investigate there.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the FreeBSD/OSX man page:

Open file descriptors are now passed as ancillary data for AF_UNIX domain
sockets, with cmsg_level set to SOL_SOCKET and cmsg_type set to
SCM_RIGHTS.

I am not sure if they mean that is done automatically or that you should do that.

for cmsg_level, cmsg_type, cmsg_data in ancdata:
if (cmsg_level == _socket.SOL_SOCKET and cmsg_type == _socket.SCM_RIGHTS):
fds.frombytes(cmsg_data[:
len(cmsg_data) - (len(cmsg_data) % fds.itemsize)])

return msg, list(fds), flags, addr
__all__.append("recv_fds")

if hasattr(_socket.socket, "share"):
def fromshare(info):
""" fromshare(info) -> socket object
Expand Down
45 changes: 44 additions & 1 deletion Lib/test/test_socket.py
100644 → 100755
Original file line number Diff line number Diff line change
Expand Up @@ -6186,11 +6186,53 @@ def test_dual_stack_client_v6(self):
self.echo_server(sock)
self.echo_client(("::1", port), socket.AF_INET6)

@requireAttrs(socket, "send_fds")
@requireAttrs(socket, "recv_fds")
@requireAttrs(socket, "AF_UNIX")
class SendRecvFdsTests(unittest.TestCase):
def testSendAndRecvFds(self):
def close_pipes(pipes):
for fd1, fd2 in pipes:
os.close(fd1)
os.close(fd2)

def close_fds(fds):
for fd in fds:
os.close(fd)

# send 10 file descriptors
pipes = [os.pipe() for _ in range(10)]
self.addCleanup(close_pipes, pipes)
fds = [rfd for rfd, wfd in pipes]

# use a UNIX socket pair to exchange file descriptors locally
sock1, sock2 = socket.socketpair(socket.AF_UNIX, socket.SOCK_STREAM)
with sock1, sock2:
socket.send_fds(sock1, [MSG], fds)
# request more data and file descriptors than expected
msg, fds2, flags, addr = socket.recv_fds(sock2, len(MSG) * 2, len(fds) * 2)
self.addCleanup(close_fds, fds2)

self.assertEqual(msg, MSG)
self.assertEqual(len(fds2), len(fds))
self.assertEqual(flags, 0)
# don't test addr

# test that file descriptors are connected
for index, fds in enumerate(pipes):
rfd, wfd = fds
os.write(wfd, str(index).encode())

for index, rfd in enumerate(fds2):
data = os.read(rfd, 100)
self.assertEqual(data, str(index).encode())


def test_main():
tests = [GeneralModuleTests, BasicTCPTest, TCPCloserTest, TCPTimeoutTest,
TestExceptions, BufferIOTest, BasicTCPTest2, BasicUDPTest,
UDPTimeoutTest, CreateServerTest, CreateServerFunctionalTest]
UDPTimeoutTest, CreateServerTest, CreateServerFunctionalTest,
SendRecvFdsTests]

tests.extend([
NonBlockingTCPTests,
Expand Down Expand Up @@ -6254,5 +6296,6 @@ def test_main():
support.run_unittest(*tests)
support.threading_cleanup(*thread_info)


if __name__ == "__main__":
test_main()
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
The socket module now has the :func:`socket.send_fds` and :func:`socket.recv.fds` methods.
Contributed by Joannah Nanjekye, Shinya Okano and Victor Stinner.