asyncio shell subprocess does not close properly

[abauske]

Environment:
Windows 10, Python 3.9.0

consider this example script:

#!/usr/bin/env python3
import asyncio

async def execAsync():
    cmd = 'timeout 10'
    process = await asyncio.create_subprocess_shell(cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.STDOUT)
    try:
        while process.returncode is None:
            output = await process.stdout.readline()
            print(output)
            await asyncio.sleep(0)  # in case of a continuous streem of b'' give other processes time to do their stuff
        await process.wait()  # this seems to do nothing of help
    except asyncio.CancelledError:
        process.terminate()  # this seems to do nothing of help
        await process.wait()  # this seems to do nothing of help
    finally:
        process.terminate()  # this seems to do nothing of help
        # process._transport.close()  # This is required for me. "ValueError: I/O operation on closed pipe" otherwise
        await process.wait()  # this seems to do nothing of help

async def main():
    canReader = asyncio.create_task(execAsync())
    await asyncio.sleep(1)
    canReader.cancel()
    await canReader
    return 0

if __name__ == '__main__':
    exit(asyncio.run(main()))

with process._transport.close() commented (as above) results in the following error:

Expand

Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x0000018B72FF4A60>
Traceback (most recent call last):
File "<path to user folder>\AppData\Local\Programs\Python\Python39\lib\asyncio\proactor_events.py", line 115, in __del__
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
File "<path to user folder>\AppData\Local\Programs\Python\Python39\lib\asyncio\proactor_events.py", line 79, in __repr__
  info.append(f'fd={self._sock.fileno()}')
File "<path to user folder>\AppData\Local\Programs\Python\Python39\lib\asyncio\windows_utils.py", line 102, in fileno
  raise ValueError("I/O operation on closed pipe")
ValueError: I/O operation on closed pipe
Exception ignored in: <function BaseSubprocessTransport.__del__ at 0x0000018B72FEE160>
Traceback (most recent call last):
File "<path to user folder>\AppData\Local\Programs\Python\Python39\lib\asyncio\base_subprocess.py", line 125, in __del__
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
File "<path to user folder>\AppData\Local\Programs\Python\Python39\lib\asyncio\base_subprocess.py", line 78, in __repr__
  info.append(f'stdout={stdout.pipe}')
File "<path to user folder>\AppData\Local\Programs\Python\Python39\lib\asyncio\proactor_events.py", line 79, in __repr__
  info.append(f'fd={self._sock.fileno()}')
File "<path to user folder>\AppData\Local\Programs\Python\Python39\lib\asyncio\windows_utils.py", line 102, in fileno
  raise ValueError("I/O operation on closed pipe")
ValueError: I/O operation on closed pipe

with process._transport.close() it finishes without an error. but _transport seems to not be made to be accessed. Is there something I have missed to fix this?

See this Stackoverflow where I have found that solution (and decided to create this issue as this seems not intended).

[gvanrossum]

@knight37x Did you perhaps run a slightly different program than what you pasted above? When I copy-paste that into a file named t.py and run python3.10 t.py I get an endless stream of

b''

When I type the ssh command in the command line, I get an immediate error:

$ ssh [email protected]
ssh: connect to host 127.0.0.1 port 22: Connection refused
$ 

When I replace the ssh command in the program with sleep 2 I get a ProcessLookupError exception.

All this is on macOS, but that should hardly matter.

What should I do

[gvanrossum]

All this is on macOS, but that should hardly matter.

Whoops, famous last words. :-) The issue does appear on Windows with Python 3.10.

However, with Python 3.11.1 or 3.12a2+ I get the same ProcessLookupError exception that I saw on macOS.

The ssh command gives the same error for me on Windows.

Can you reduce your example to something that doesn't depend on the behavior of ssh?

[abauske]

@gvanrossum I found out you can replace that ssh command by e.g. timeout 10 to get the same I/O operation on closed pipe error. I also added an additional await asyncio.sleep(0) in while loop to give "main" CPU time.
I updated the script up above accordingly. Could you please try this version? Is that what you were looking for?

[kumaraditya303]

I found out you can replace that ssh command by e.g. timeout 10 to get the same I/O operation on closed pipe error.

That issue is already fixed with #88050. You should try 3.11.1 and current main.

[abauske]

oh nice! sorry for the noise!