Skip to content

bpo-28724: Add methods send_fds, recv_fds to the socket module #12889

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 27 commits into from
Sep 11, 2019
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
fd7b901
Add methods send_fds, recv_fds to the socket module
nanjekyejoannah May 30, 2019
2558c5d
📜🤖 Added by blurb_it.
blurb-it[bot] May 30, 2019
dfdeb76
Fix tests on MacOS
nanjekyejoannah May 30, 2019
bc58424
Merge branch 'issue28724' of https://github.com/nanjekyejoannah/cpyth…
nanjekyejoannah May 30, 2019
cc6973c
Attempt to fix MacOS
nanjekyejoannah May 30, 2019
5050e59
MacOS fix
nanjekyejoannah May 30, 2019
fe8ff1a
Mac OS fix
nanjekyejoannah May 30, 2019
e98918f
fix Mac OS
nanjekyejoannah May 30, 2019
c3fcd6f
Attempt again
nanjekyejoannah May 30, 2019
d497c15
keep the function signatures close to the original ones
nanjekyejoannah May 31, 2019
035845a
make return types consistent
nanjekyejoannah May 31, 2019
e6d22e0
Fix freebsd
nanjekyejoannah Jun 1, 2019
f84d6db
Remove unused global sentinel .
nanjekyejoannah Jun 3, 2019
07b05dc
Update documentation
nanjekyejoannah Jun 6, 2019
8b96493
Fix docs
nanjekyejoannah Jun 6, 2019
7c46e7a
Update Doc/library/socket.rst
nanjekyejoannah Jun 6, 2019
2c32da1
Add Victor's test and update docs
Jun 13, 2019
75bffbc
no need for new file
nanjekyejoannah Jun 13, 2019
ba29ca2
Update 3.8.rst
nanjekyejoannah Jun 13, 2019
83e422d
update 3.9.rst
Jun 13, 2019
910461b
Delete 3.9.rst
nanjekyejoannah Jun 13, 2019
b134b48
Update test_socket.py
nanjekyejoannah Jun 13, 2019
ee950f8
Update test_socket.py
nanjekyejoannah Jun 13, 2019
36e157d
fix space
Jun 13, 2019
0f03c42
Do not skip Macos in tests
nanjekyejoannah Aug 24, 2019
bf67bd8
Fix doc strings
nanjekyejoannah Aug 28, 2019
1ac7fbf
Fix test
nanjekyejoannah Sep 11, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions Doc/library/socket.rst
Original file line number Diff line number Diff line change
Expand Up @@ -669,6 +669,30 @@ The following functions all create :ref:`socket objects <socket-objects>`.
.. versionadded:: 3.3


.. function:: socket.send_fds(sock, buffers, fds[, flags[, address]])

:func:`socket.send_fds` sends the list of file descriptors *fds*
over an :const:`AF_UNIX` socket, on systems which support the
:const:`SCM_RIGHTS` mechanism.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are buffers, flags and address? Maybe suggest to read sendmsg() for the documentation of these parameters?


.. availability:: most Unix platforms, possibly others.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest "availability: Unix supporting :meth:socket.socket.sendmsg and :const:SCM_RIGHTS mechanism" (and remove "possibly others").


.. versionadded:: 3.8


.. function:: socket.recv_fds(sock, bufsize, maxfds[, flags])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is bufsize? How should I use it? What is flags? Maybe suggest to read recvmsg() for the documentation of flags?


On systems which support the :const:`SCM_RIGHTS` mechanism,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest "availability: Unix supporting :meth:socket.socket.recvmsg and :const:SCM_RIGHTS mechanism" (and remove "possibly others"). Usually, we don't start a doc by documenting the availability.

:func:`socket.recv_fds` will receive up to *maxfds* file descriptors,
returning the message data and a list containing the descriptors
(while ignoring unexpected conditions such as unrelated control
messages being received).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest:

Receive up to maxfds file descriptors.
Return (msg_data, fds) where msg_data is the message data and fds is a list of file descriptors.

It seems like it returns up to 4 items.

I don't understand "(while ignoring unexpected conditions such as unrelated control messages being received)." What is the behavior in case of "unrelated control messages being received"? What is the return format?


.. availability:: most Unix platforms, possibly others.

.. versionadded:: 3.8
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's too late for 3.8, you should now target 3.9.



.. data:: SocketType

This is a Python type object that represents the socket object type. It is the
Expand Down
5 changes: 5 additions & 0 deletions Doc/whatsnew/3.8.rst
Original file line number Diff line number Diff line change
Expand Up @@ -560,6 +560,11 @@ The :func:`socket.if_nameindex()`, :func:`socket.if_nametoindex()`, and
:func:`socket.if_indextoname()` functions have been implemented on Windows.
(Contributed by Zackery Spytz in :issue:`37007`.)

Added :func:`socket.send_fds()` and :func:`socket.recv_fds` to send and
receive a file descriptor from the socket.
(Contributed by Joannah Nanjekye in :issue:`28724`.)


shlex
----------

Expand Down
32 changes: 31 additions & 1 deletion Lib/socket.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@
socket() -- create a new socket object
socketpair() -- create a pair of new socket objects [*]
fromfd() -- create a socket object from an open file descriptor [*]
send_fds() -- Send file descriptor to the socket.
recv_fds() -- Recieve file descriptor from the socket.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest to replace "file descriptor" with "file descriptors" for both functions here (add an S), as in their name, since they can send and receive more than one file descriptor ;-)

fromshare() -- create a socket object from data received from socket.share() [*]
gethostname() -- return the current hostname
gethostbyname() -- map a hostname to its IP number
Expand Down Expand Up @@ -49,7 +51,7 @@
import _socket
from _socket import *

import os, sys, io, selectors
import os, sys, io, selectors, array
from enum import IntEnum, IntFlag

try:
Expand Down Expand Up @@ -463,6 +465,34 @@ def fromfd(fd, family, type, proto=0):
nfd = dup(fd)
return socket(family, type, proto, nfd)

if hasattr(_socket.socket, "sendmsg"):
def send_fds(sock, buffers, fds, flags=0, address=None):
""" send_fds(sock, buffers, fds[, flags[, address]]) -> socket object
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like sendmsg() returns an integer, not a socket object.


Send the list of file descriptors fds over an AF_UNIX socket.
"""
return sock.sendmsg(buffers, [(_socket.SOL_SOCKET,
_socket.SCM_RIGHTS, array.array("i", fds))])
__all__.append("send_fds")

if hasattr(_socket.socket, "recvmsg"):
def recv_fds(sock, bufsize, maxfds, flags=0):
""" recv_fds(sock, bufsize, maxfds[, flags]) -> (socket object, socket object)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The doc of the return value looks wrong. The code uses a tuple of 4 times: return msg, list(fds), flags, addr.


receive up to maxfds file descriptors returning the message
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Upper case for the first word of the doc?

Suggested change
receive up to maxfds file descriptors returning the message
Receive up to maxfds file descriptors returning the message

data and a list containing the descriptors.
"""
# Array of ints
fds = array.array("i")
msg, ancdata, flags, addr = sock.recvmsg(bufsize,
_socket.CMSG_LEN(maxfds * fds.itemsize))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will likely fail on FreeBSD as RFC 3542 requires portable applications to use CMSG_SPACE if I am not mistaken.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should check that this works on FreeBSD to avoid broken buildbots with obscure errors.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pablogsal You mean at test level ?

Copy link
Member

@pablogsal pablogsal Jun 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean that FreeBSD systems running that code can fail and that will likely appear when running the tests that go over this code path.

This is an example of similar scenarios of CMSG_LEN vs CMSG_SPACE:

#9613

I could be wrong with my argument, that's why I think we should check directly on a FreeBSD system.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont have a freeBSD system to confirm this atm. When I do find, I will confirm but someone else can help confirm this. Do you have a FreeBSD to help check this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

@pablogsal pablogsal Jun 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that my suspicions were correct:

test_errors (test.test_socket.SendfileUsingSendfileTest) ... ok
test test_socket failed
test_new_tcp_flags (test.test_socket.TestMSWindowsTCPFlags) ... skipped 'requires Windows'
======================================================================
ERROR: testSendAndRecvFds (test.test_socket.BasicTCPTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/custom.koobs-freebsd10.nondebug/build/Lib/test/test_socket.py", line 2294, in testSendAndRecvFds
    socket.send_fds(sock, [MSG], fds)
  File "/usr/home/buildbot/python/custom.koobs-freebsd10.nondebug/build/Lib/socket.py", line 476, in send_fds
    return sock.sendmsg(buffers, [(_socket.SOL_SOCKET,
OSError: [Errno 22] Invalid argument
======================================================================
ERROR: testSendAndRecvFds (test.test_socket.BasicTCPTest2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/custom.koobs-freebsd10.nondebug/build/Lib/test/test_socket.py", line 2294, in testSendAndRecvFds
    socket.send_fds(sock, [MSG], fds)
  File "/usr/home/buildbot/python/custom.koobs-freebsd10.nondebug/build/Lib/socket.py", line 476, in send_fds
    return sock.sendmsg(buffers, [(_socket.SOL_SOCKET,
OSError: [Errno 22] Invalid argument
----------------------------------------------------------------------
Ran 581 tests in 29.349s

but it could be something also on top of CMSG_LEN

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@koobs Could you please take a look? The new feature is failing on FreeBSD.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can confirm that is the same error as in OSX so it may be easier to investigate there.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the FreeBSD/OSX man page:

Open file descriptors are now passed as ancillary data for AF_UNIX domain
sockets, with cmsg_level set to SOL_SOCKET and cmsg_type set to
SCM_RIGHTS.

I am not sure if they mean that is done automatically or that you should do that.

for cmsg_level, cmsg_type, cmsg_data in ancdata:
if (cmsg_level == _socket.SOL_SOCKET and cmsg_type == _socket.SCM_RIGHTS):
# Append data, ignoring any truncated integers at the end.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we document this in the recv_fds() doc?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but IMHO it's worth it to mention that "any truncated integers at the end are ignored".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, another solution is to raise an hard error rather than silently ignoring a potential bug.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nanjekyejoannah: Would you mind to document this behavior in recv_fds() documention?

fds.fromstring(cmsg_data[:len(cmsg_data) - (len(cmsg_data) % fds.itemsize)])
return msg, list(fds), flags, addr
__all__.append("recv_fds")

if hasattr(_socket.socket, "share"):
def fromshare(info):
""" fromshare(info) -> socket object
Expand Down
25 changes: 25 additions & 0 deletions Lib/test/test_socket.py
Original file line number Diff line number Diff line change
Expand Up @@ -2275,6 +2275,31 @@ def testFromFd(self):
def _testFromFd(self):
self.serv_conn.send(MSG)

@requireAttrs(socket.socket, "sendmsg")
@requireAttrs(socket, "AF_UNIX")
@unittest.skipIf(sys.platform == 'darwin', 'test not working for MacOSX')
@unittest.skipIf(sys.platform.startswith(('freebsd', 'netbsd', 'gnukfreebsd')), 'test not working for freebsd')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait, why are we ignoring the test in FreeBSD? We should fix the code/test instead, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixing means I remove those parameters that freebsd complains about in the function itself. Is that the right fix? and will require changing the docs too to say so. My assumption is that freebsd doesn't support those attributes we are calling that is why it is failing.

Copy link
Member

@pablogsal pablogsal Jun 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FREEBSD supports those functions (sendmsg and recvmsg) with SCM_RIGHTS:

https://www.freebsd.org/cgi/man.cgi?query=recvmsg&sektion=2&manpath=FreeBSD+6.0-RELEASE

As well as MacOSX:

https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/recvmsg.2.html

I don't think we should ignore the test in those systems. We should understand why both systems complain with the current implementation. Maybe I am missing something, though.

Copy link
Contributor Author

@nanjekyejoannah nanjekyejoannah Jun 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error returned is general pointing to invalid attributes. It may not be SCM_RIGHTS but may be it is SOL_SOCKET that is also used in the definition of send_fds() in this case.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SOL_SOCKET is supported in both platforms (check the man pages for reference) but clearly there is something wrong with either the combination or that SCM_RIGHTS should be handled differently on both platforms, but IMHO skipping the test is not the correct thing to do here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree finding a remedy is better. I may have time next week maybe to simulate this freebsd thing to troubleshoot this but If anyone wants to jump in now, no problem :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will try to find some time this week to help you in investigating this :)

def testSendAndRecvFds(self):
fds = []
# create two new file descriptors.
for i in range(2):
fd, path = tempfile.mkstemp()
self.addCleanup(os.unlink, path)
self.addCleanup(os.close, fd)
os.write(fd, str(i).encode())
fds.append(fd)
f = self.cli_conn.detach()
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM, fileno=f)
self.addCleanup(sock.close)
with sock:
socket.send_fds(sock, [MSG], fds)
msg, fds_list, flags, addr = socket.recv_fds(sock, len(MSG), 1024)
self.assertEqual(msg, MSG)

@testSendAndRecvFds.client_skip
def _testSendAndRecvFds(self):
self.serv_conn.send(MSG)

def testDup(self):
# Testing dup()
sock = self.cli_conn.dup()
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
The socket module now has the :func:`socket.send_fds()` and :func:`socket.recv.fds()` functions.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a method, not a function, no? You should omit parenthesis.

The :class:socket.socket type gets 2 new methods: :meth:socket.socket.send_fds and :meth:socket.socket.recv.fds.

Contributed by Joannah Nanjekye and Shinya Okano .